Kafka and Spark Streaming Interview Questions and Answers

Kafka interviews test whether you understand how a distributed log works, and whether you can keep data correct when things fail. This guide gives you 52 Kafka and Spark Streaming interview questions and answers, from basics to scenario questions, for freshers and experienced data engineers.

Every answer starts with a short, direct response you can say aloud, followed by a table, a setting or a code sample where it helps. The page reflects current Kafka. It treats ZooKeeper as history, explains KRaft and the new consumer group protocol, and covers Spark Structured Streaming rather than the legacy DStream API.

The guide has eight parts: 8 on Kafka basics, 7 on producers, 8 on consumers and offsets, 8 on replication and reliability, 5 on KRaft and operations, 4 on Kafka Connect, Streams and comparisons, 6 on Spark Structured Streaming with Kafka and 6 scenario questions. Use the contents list to jump to your level.

Key points

• 52 questions in eight parts, from Kafka basics to scenarios.
• Current for Kafka 4.x: KRaft only, no ZooKeeper.
• Delivery guarantees, consumer lag and rebalancing explained with settings.
• Spark Structured Streaming with Kafka, including checkpoints and exactly once.
• Comparison tables for Kafka against Flume, RabbitMQ and Kafka Streams.

In this guide

Kafka Basics (Freshers)

Start with what Kafka is for and the handful of terms every other answer builds on: topic, partition, offset, broker and consumer group.

Q1. What is Apache Kafka, and what is it used for?

Apache Kafka is a distributed event streaming platform that stores streams of records durably and lets many applications publish and read them in real time. It is used for log and metrics pipelines, event driven microservices, change data capture, stream processing and feeding data lakes. Kafka is written in Java and Scala.

Say that Kafka is a durable, replayable log, not a queue that deletes a message once it is read. That one idea separates strong answers from average ones.

Q2. What are a topic, a partition and an offset?

A topic is a named stream of records. Each topic is split into partitions, and each partition is an ordered, append only log. An offset is the position of a record inside one partition. Order is guaranteed only within a partition, never across the whole topic.

Q3. What are brokers, and how does a Kafka cluster work?

A broker is a Kafka server that stores partitions and serves producers and consumers. A cluster is several brokers working together. Each partition has one leader broker that handles reads and writes, and follower brokers that copy it. If a leader fails, a follower from the in sync replicas becomes the new leader.

Q4. What is a consumer group?

A consumer group is a set of consumers that share the work of reading a topic. Each partition is assigned to only one consumer in the group at a time, so adding consumers up to the partition count increases throughput. Different groups read the same topic independently, each with its own offsets.

What the interviewer is listening for: You mention that extra consumers beyond the partition count sit idle.

Q5. How is Kafka different from a traditional message queue?

A traditional queue usually deletes a message after one consumer takes it. Kafka keeps records for a retention period whatever consumers do, so many groups can read the same data and can replay it from any offset. Kafka also scales by partitioning, and it is built for very high throughput.

Q6. Does Kafka still need ZooKeeper?

No. Kafka 4.0 removed ZooKeeper completely, and a cluster now runs only in KRaft mode, where Kafka manages its own metadata through a Raft based controller quorum. Older versions needed a separate ZooKeeper ensemble. If an interviewer asks about ZooKeeper, explain that it is history and describe KRaft instead.

What the interviewer is listening for: You know the change happened in 4.0 and you do not describe ZooKeeper as a current requirement.

Q7. Where does Kafka store data, and how long does it keep it?

Each partition is stored on the broker disk as a set of segment files in a log directory. Kafka keeps records until the retention limit by time or size is reached, which defaults to seven days, or forever if you set it that way. Compacted topics instead keep the latest record for each key.

Q8. Why is Kafka fast?

Kafka writes records sequentially to disk, which is quick, and relies on the operating system page cache. It batches and compresses records, and it can send data from disk to the network with zero copy transfer. Partitioning spreads the load across brokers, so throughput grows by adding partitions and machines.

Producers (Freshers and Intermediate)

Producer questions are about how records reach the right partition, how safely, and how fast.

Q9. How does a producer choose the partition for a record?

If the record has a key, the default partitioner hashes the key so the same key always goes to the same partition, which keeps the order for that key. If there is no key, records are spread across partitions in batches. You can also set the partition directly or write a custom partitioner.

Changing the partition count of a topic changes where a key lands, so plan the count before you rely on key ordering.

Q10. What do the acks settings mean?

The acks setting controls how many acknowledgements the producer needs before a write counts as successful. With acks=0 it does not wait. With acks=1 only the leader must confirm. With acks=all the leader waits for all in sync replicas, which is the safest choice and the default in current Kafka producers.

Q11. What is an idempotent producer?

An idempotent producer stops retries from creating duplicates. Each producer gets an ID and every record carries a sequence number per partition, so the broker discards a record it has already written. It is enabled by default in current clients, and it needs acks=all.

bootstrap.servers=broker1:9092,broker2:9092
acks=all
enable.idempotence=true
linger.ms=5
compression.type=lz4

linger.ms and batch.size control batching. A small wait lets the producer fill larger batches, which raises throughput at a little cost in latency. The default linger.ms is 5 in Kafka 4.0 and later.

Q12. How do you tune a producer for throughput and for latency?

For throughput, raise batch.size and linger.ms, enable compression and keep acks at all. For latency, keep linger.ms low and batches small. Compression such as lz4 or zstd reduces network and disk use and usually helps more than it costs. Always measure with your own record sizes.

• Throughput: larger batch.size, linger.ms of 5 to 50, compression on.
• Latency: linger.ms near 0, smaller batches.
• Memory: buffer.memory limits how much the producer can hold before it blocks.

Q13. What happens if a producer cannot reach the broker?

The producer retries within delivery.timeout.ms, which defaults to two minutes. Records wait in its buffer meanwhile. If the timeout passes, the send fails and the callback receives an error that your code must handle. With idempotence on, the retries do not create duplicates.

Q14. What is the difference between synchronous and asynchronous sending?

An asynchronous send returns immediately and reports the result in a callback, which gives the best throughput. A synchronous send waits for the result before continuing, for example by calling get on the returned future, which is simpler but slow. Use asynchronous sends with a callback in almost every real application.

Q15. What is message compaction, and when would you use a compacted topic?

Log compaction keeps at least the latest record for each key and removes older ones. It suits topics that hold the current state, such as the latest profile of each customer, a changelog or a configuration table. A record with a null value, called a tombstone, marks a key for deletion.

Consumers and Offsets (Intermediate)

Consumer questions test whether you understand offsets, rebalancing and delivery guarantees, which are where most production problems appear.

Q16. How does a consumer track what it has read?

A consumer commits offsets, which record the next position to read in each partition. Kafka stores these in an internal topic called __consumer_offsets. After a restart or a rebalance, the consumer resumes from the last committed offset. Committing before processing risks loss, and committing after processing risks duplicates.

Q17. What are automatic and manual offset commits?

With automatic commits, the consumer commits on a timer set by auto.commit.interval.ms, which is simple but can lose or repeat records after a crash. With manual commits, your code commits after it has processed a batch, either synchronously or asynchronously, which gives you control over the delivery guarantee.

props.put("enable.auto.commit", "false");
props.put("group.id", "orders-service");

// inside the poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> r : records) {
    process(r);
}
consumer.commitSync();     // commit only after processing

Q18. What are at most once, at least once and exactly once delivery?

At most once means a record may be lost but is never repeated. At least once means it is never lost but may be repeated, which is the usual default. Exactly once means the effect is applied one time, which Kafka supports through idempotent producers and transactions inside Kafka, with care at the edges.

Q19. What is a consumer rebalance, and why can it hurt?

A rebalance redistributes partitions among the consumers in a group when members join, leave or fail, or when the topic changes. With the older eager protocol, all consumers stop reading while this happens, which causes lag spikes. Cooperative rebalancing and the newer consumer group protocol reduce the pause.

KIP 848, the new consumer group protocol, became generally available in Kafka 4.0. Rebalancing moves to the broker side, so a change affects only the members whose partitions move.

Q20. What causes frequent rebalances, and how do you reduce them?

The usual causes are a consumer taking longer than max.poll.interval.ms between polls, a missed heartbeat that exceeds session.timeout.ms, and frequent restarts or autoscaling. Reduce them by keeping processing per poll short, lowering max.poll.records, tuning the timeouts, using static membership and moving to the newer protocol.

• Lower max.poll.records if each batch takes too long.
• Raise max.poll.interval.ms only if the slow work is genuine.
• Set group.instance.id for static membership so a quick restart does not trigger a rebalance.

Q21. What is consumer lag, and how do you monitor it?

Consumer lag is the difference between the latest offset in a partition and the group committed offset, which shows how far behind a consumer is. Growing lag means consumers are too slow or stopped. Check it with the kafka consumer groups command or a monitoring tool, and alert on a trend, not one value.

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group orders-service

The output shows the current offset, the log end offset and the lag for each partition. A lag that rises on one partition only often points to a hot key.

Q22. How do you replay data or reset offsets?

Because Kafka keeps records, a group can reread them. Stop the consumers, then use the consumer groups tool with the reset offsets option to move to the earliest offset, a timestamp or a specific offset, and start again. In code, you can call seek on a partition. Always run the dry run first.

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --group orders-service --topic orders \
  --reset-offsets --to-earliest --dry-run

Q23. What is the difference between a consumer group and a share group?

A classic consumer group assigns each partition to one consumer, so parallelism is capped by the partition count. A share group, introduced in KIP 932 and production ready in Kafka 4.2, lets several consumers read records from the same partitions with per record acknowledgement, which gives queue like behaviour on a Kafka topic.

What the interviewer is listening for: You show awareness that Kafka now supports queue style consumption, and you can say when each model fits.

Replication and Reliability (Intermediate and Experienced)

Experienced interviews focus on how Kafka survives failures and what settings protect your data.

Q24. What is the replication factor, and what are in sync replicas?

The replication factor is the number of copies of each partition, commonly 3 in production. The in sync replica set, or ISR, is the leader plus the followers that are fully caught up. Only members of the ISR can become leader without data loss, and acks=all waits for the whole ISR.

Q25. What does min.insync.replicas do?

It sets the minimum number of in sync replicas that must acknowledge a write when acks=all. With a replication factor of 3 and min.insync.replicas of 2, writes continue if one broker fails and are rejected if two fail, which protects data. Setting it to 1 removes that protection.

Q26. What is unclean leader election?

Unclean leader election allows a replica that is not in sync to become leader when no in sync replica is available. This restores availability but can lose records. It is off by default. Turn it on only for a topic where availability matters more than losing some data, and say so openly.

Q27. How do you choose the number of partitions for a topic?

Work from your target throughput and your consumer count. Divide the throughput you need by the measured throughput of one partition for producers and consumers, and take the larger result. Allow room for growth, since the count can be raised but not lowered. Too many partitions add memory use and slower recovery.

A common starting point is to match the maximum number of consumers you expect in the largest group. Test with your real message size before settling.

Q28. How does Kafka guarantee ordering?

Kafka guarantees order only within a single partition. To keep related events in order, give them the same key so they land in the same partition. For strict ordering with retries, keep idempotence on, which preserves order for up to five in flight requests per connection.

Q29. How do Kafka transactions work?

A producer with a transactional ID can write to several partitions and topics atomically, and also commit consumed offsets as part of the same transaction. Consumers set isolation.level to read_committed so they see only committed records. This gives exactly once processing for a read, process and write pipeline inside Kafka.

producer.initTransactions();
try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("output", key, value));
    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
} catch (KafkaException e) {
    producer.abortTransaction();
}

Q30. What is the difference between log retention and log compaction?

Retention deletes whole segments once they are older than the time limit or the topic exceeds the size limit. Compaction instead keeps the latest record for every key and removes older versions, with no time limit. A topic can use both by setting cleanup.policy to compact,delete.

Q31. How do you secure a Kafka cluster?

Encrypt traffic with TLS, authenticate clients with SASL such as SCRAM or OAuth, or with mutual TLS, and authorise them with access control lists on topics, groups and cluster actions. Keep inter broker traffic on a protected listener, and rotate credentials. Never leave a production listener as plaintext.

KRaft, Operations and Upgrades (Experienced)

These questions check whether your knowledge is current. Kafka changed a lot between versions 3 and 4.

Q32. What is KRaft, and how does it replace ZooKeeper?

KRaft is Kafka’s built in consensus layer based on Raft. A small quorum of controller nodes stores cluster metadata in an internal log called the metadata topic, and elects an active controller. It removes the separate ZooKeeper system, scales to more partitions and makes failover and startup faster.

Q33. What runtime does current Kafka need?

Kafka 4.x brokers, controllers and Connect need Java 17, while the clients and Kafka Streams need Java 11. Check the version table in the official documentation before an upgrade, because requirements change between releases.

What the interviewer is listening for: You know that Java requirements are different for brokers and clients.

Q34. How do you upgrade from Kafka 3 to Kafka 4?

A cluster that still uses ZooKeeper must first be migrated to KRaft, using a supported 3.x release, before moving to 4.0. Then do a rolling upgrade one broker at a time, and finalise the metadata version at the end. Read the upgrade notes for removed features such as MirrorMaker 1 and old protocol versions.

Test the whole path in a staging cluster, and keep clients on supported versions. This is a good place to mention a rollback plan.

Q35. How do you replicate data between clusters?

Use MirrorMaker 2, which is built on Kafka Connect and copies topics, consumer group offsets and configuration between clusters. MirrorMaker 1 was removed in Kafka 4.0. Typical uses are disaster recovery, moving to a new cluster and bringing regional data to a central cluster.

Q36. Which metrics do you watch on a Kafka cluster?

Watch under replicated partitions, offline partitions, active controller count, request latency, disk usage, network throughput, and consumer lag. Under replicated partitions that stay above zero signal a failing or slow broker, and lag that keeps growing signals a slow consumer. Alert on trends and on any offline partition.

Kafka Connect, Kafka Streams and Comparisons (Intermediate)

Interviewers like to see that you know where Kafka sits next to other tools.

Q37. What is Kafka Connect?

Kafka Connect is a framework for moving data in and out of Kafka without writing custom code. Source connectors pull data from systems such as databases into topics, and sink connectors push topic data to systems such as object storage or a search index. It handles scaling, offsets and failure recovery for you.

Q38. What is Kafka Streams, and how does it differ from Spark Streaming?

Kafka Streams is a Java library you embed in your own application to process Kafka topics, with no separate cluster. Spark Structured Streaming runs on a Spark cluster and suits large analytics and joins with other data sources. Choose Streams for lightweight microservice processing and Spark for heavy analytics.

Q39. What is the difference between Kafka and Flume?

Kafka is a durable, replayable event log that many consumers can read, and it suits event streaming at scale. Flume is a tool for collecting log data into Hadoop with sources, channels and sinks, and it has been dormant since 2024, with no new releases. For new designs, choose Kafka.

If the role still involves an old Flume pipeline, say you would move it to Kafka with Connect or a lightweight collector when the work is justified.

Q40. What is the difference between Kafka and RabbitMQ?

RabbitMQ is a broker built around routing, exchanges and per message acknowledgement, and it removes messages once they are consumed. Kafka is a partitioned log built for high throughput, replay and stream processing. Pick RabbitMQ for flexible routing and task queues, and Kafka for event streaming and data pipelines.

Spark Structured Streaming with Kafka (Intermediate and Experienced)

Many interviews pair Kafka with Spark. These questions cover how Spark reads from Kafka and how it keeps results correct. DStreams are legacy, so answer with Structured Streaming.

Q41. What is Spark Structured Streaming?

Structured Streaming is the Spark engine for stream processing built on the DataFrame and SQL API. You write a query as if the data were a static table, and Spark runs it incrementally as new data arrives. It gives fault tolerance and supports event time, windows and joins. The old DStream API is legacy.

Q42. How do you read from Kafka in Structured Streaming?

Use the Kafka source with the readStream call, set the bootstrap servers and the topic, and add the spark sql kafka connector package that matches your Spark and Scala version. The result has binary key and value columns, which you cast to strings and then parse. Spark 4 supports Scala 2.13 only.

df = (spark.readStream.format("kafka")
      .option("kafka.bootstrap.servers", "broker1:9092")
      .option("subscribe", "orders")
      .option("startingOffsets", "latest")
      .load())

orders = df.selectExpr("CAST(key AS STRING) AS key", "CAST(value AS STRING) AS value", "timestamp")

The Kafka source also gives you topic, partition, offset and timestamp columns, which are useful for debugging.

Q43. Where does Spark keep Kafka offsets, and how does it recover?

Spark does not rely on the Kafka consumer group commit. It records the offsets it has processed in the checkpoint location, along with the query state. After a failure, the query restarts from the checkpoint and reprocesses from the stored offsets. This is why every production query needs a durable checkpoint path.

What the interviewer is listening for: You say the offsets are in the checkpoint, not in Kafka consumer group metadata.

Q44. How do you get exactly once results from Spark and Kafka?

Spark gives exactly once semantics for processing when the source is replayable, like Kafka, the checkpoint is durable and the sink is idempotent or transactional. Writing to Kafka is at least once by default, so deduplicate downstream, or write to a sink that supports upserts, such as a Delta table with a merge.

Q45. What are watermarks and windows?

A window groups events by event time, such as every five minutes. A watermark tells Spark how late data may arrive, so it can finish a window and discard old state. Without a watermark, state for aggregations grows without limit. Data later than the watermark may be dropped.

from pyspark.sql.functions import window, col

counts = (orders.withWatermark("timestamp", "10 minutes")
                .groupBy(window(col("timestamp"), "5 minutes"))
                .count())

Q46. How do you control the rate of reading from Kafka in Spark?

Set maxOffsetsPerTrigger to limit how many records one micro batch reads, and choose a trigger interval to control how often a batch runs. This protects the cluster after a backlog, when a restart could otherwise try to read hours of data at once. Monitor batch duration against the trigger interval.

Scenario Questions (Experienced)

Scenario questions test how you reason under pressure. State your assumptions, the checks you run first and the fix.

Q47. Consumer lag on one topic keeps growing. How do you investigate?

First see whether lag is growing on all partitions or only some. If all, consumers are too slow or too few, so check processing time, scale consumers up to the partition count and look at downstream calls. If only some partitions lag, suspect a hot key or a stuck consumer. Also check for rebalance loops in the logs.

• Run the describe command and compare lag per partition.
• Check max.poll.interval.ms warnings and rebalance frequency.
• Look at the slowest step in processing, often a database or API call.

What the interviewer is listening for: You split the problem by partition first, and you suspect a hot key when lag is uneven.

Q48. A downstream system received duplicate records. What could cause it and how do you fix it?

Duplicates usually come from at least once delivery: a producer retry without idempotence, a consumer that processed a record but crashed before committing, or a rebalance. Turn on the idempotent producer, commit after processing, and make the sink idempotent with a unique key or an upsert. Add transactions where processing stays inside Kafka.

Q49. You need to guarantee no data loss for payment events. Which settings do you choose?

Use a replication factor of 3, min.insync.replicas of 2, acks=all, idempotence on, and unclean leader election off. Spread replicas across racks or zones, and set retention long enough to replay. On the consumer side commit after processing and make the handler idempotent. Test a broker failure to prove it.

Q50. How would you design a pipeline that moves database changes into a data lake?

Use change data capture with a source connector, such as Debezium on Kafka Connect, to write each change into a topic keyed by the primary key. A sink connector or a Spark Structured Streaming job then writes the topic to the lake in a table format that supports upserts. Keep the schema in a registry.

What the interviewer is listening for: You mention keys, schema management and idempotent upserts into the lake.

Q51. A topic has 6 partitions, and you add 10 consumers to the group. What happens?

Only six consumers receive partitions, since each partition goes to one consumer in the group. The other four sit idle, so throughput does not rise. To use more consumers, raise the partition count, accepting that key placement changes, or use a share group if queue style consumption fits the work.

Q52. One broker has run out of disk space. What do you do?

First stop the growth: check which topics are largest and lower their retention if it is safe, or add disks. Do not delete segment files by hand. Then rebalance partitions across brokers with the reassignment tool or a balancing tool, and add alerting on disk usage at a safe threshold, so this does not repeat.

Common questions about Kafka interviews

What topics should I prepare for a Kafka interview?

Prepare topics, partitions, offsets, consumer groups, replication and delivery guarantees first. Then learn producer and consumer settings, rebalancing, KRaft and how to monitor lag. For data engineering roles add Kafka Connect and Spark Structured Streaming. Be ready for scenario questions on lag, duplicates and data loss.

Is Kafka knowledge enough for a streaming data engineer role?

It is the core, but employers also expect a processing engine such as Spark or Flink, SQL, a programming language, and a storage layer such as a data lake. Show one end to end project that reads from Kafka, transforms the data and writes it to a table.

Which Kafka version should I learn?

Learn the current 4.x line, which runs only in KRaft mode and has the new consumer group protocol. Many companies still run older 3.x clusters, so know the ZooKeeper history well enough to explain why it was removed and how a migration works.

Do I need Java to work with Kafka?

Kafka itself runs on Java, but you can use it from Python, Scala, Go and other languages through client libraries. Java helps for Kafka Streams and for reading the documentation. Many data engineers use Python with PySpark and a client library.

How can I practise Kafka on my own laptop?

Run a single node KRaft cluster from the official download or a container, create a topic, and use the console producer and consumer. Then write a small producer and consumer in your language, simulate a failure by stopping a consumer, and watch the lag and rebalance behaviour.

A four week plan to prepare

Use this plan if you have about a month before your interview.

Practise with a trainer

If you want to practise these questions with a trainer and work on hands on projects, you can book a free demo class for the Prwatech Big Data course. It covers Kafka, Spark streaming, Hadoop and hands on projects.

Contact the learning team for current batch dates and fees.

Written by Mithun Ray, data engineering trainer at Prwatech. Reviewed by Avishekh Verma. Last updated October,2026.