Top Flume Interview Questions and Answers

Flume interviews appear mostly for roles that maintain Hadoop log pipelines. They test whether you understand sources, channels and sinks, what Flume does and does not guarantee, and whether you know what to use instead today. This guide gives you 39 Flume interview questions and answers, from basics to scenario questions.

Every answer starts with a short, direct response you can say aloud, followed by a table, a configuration or a calculation. The page is honest about the status of Flume, which has been mostly dormant since late 2024, and includes a section on migrating to Kafka and other modern tools.

The guide has six parts: 8 on Flume basics, 9 on sources, channels, sinks and interceptors, 6 on reliability and transactions, 6 on configuration and operations, 5 on Flume against modern tools and 5 scenario questions. For the modern replacement, read the Kafka and Spark Streaming interview page.

Key points

• 39 questions in six parts, from basics to scenarios.
• Honest status: Flume 1.11.0, mostly dormant since late 2024.
• Sources, channels, sinks, selectors and interceptors in tables.
• Reliability explained as at least once delivery, with the cases of loss and duplication.
• A section on what to use instead, and how to migrate to Kafka.

In this guide

Flume Interview Questions for Freshers

Start with what Flume does and where it fits. Be honest about its status, since interviewers for legacy Hadoop roles often ask whether you would choose it today.

Q1. What is Apache Flume, and what is it used for?

Apache Flume is a distributed, reliable service for collecting, aggregating and moving large amounts of log and event data from many sources into a central store such as HDFS. It is configured through a properties file and built around agents that pass events from a source through a channel to a sink. It was widely used to feed Hadoop clusters with logs.

Q2. What is the current status of Flume?

Flume is a mature Apache project, with 1.11.0 as the latest release, but it has been mostly dormant since late 2024, with very little development. It still runs in many existing Hadoop pipelines. For new projects, teams usually choose Kafka with Kafka Connect, or a lightweight log shipper. Say so clearly in an interview.

What the interviewer is listening for: You describe the status honestly, and you can explain what you would use for a new project.

Q3. What is an event in Flume?

An event is the unit of data that Flume moves. It has a body, which is a byte array holding the payload such as one log line, and optional headers, which are key value pairs holding metadata such as a timestamp or host name. Interceptors and selectors can read and change headers while an event is moving.

Q4. What is a Flume agent?

An agent is a Java process that hosts the components that move events: sources, channels and sinks. One agent can run several flows. A simple flow has one source, one channel and one sink. Agents can be chained together so that one agent sends events to the next, which is called a multi hop flow.

Q5. What are the main features and advantages of Flume?

Flume is reliable, since it uses transactions between components, scalable, since you can add agents, and flexible, since it supports many sources and sinks, multi hop, fan in and fan out flows. It needs no code for common cases, only configuration. Its drawbacks are weak ordering and delivery guarantees compared with Kafka, and no replay.

Q6. What is the difference between Flume and Sqoop?

Flume collects streaming data such as logs and events from many sources into Hadoop. Sqoop transferred bulk data between relational databases and Hadoop in batches. Sqoop has been retired to the Apache Attic. For database transfers today, use Spark JDBC reads, Kafka Connect or a change data capture tool.

Q7. What is the difference between Flume and Kafka?

Flume is a data collection pipeline of sources, channels and sinks that pushes data to a destination, with no replay and with limited consumers. Kafka is a distributed, durable log where many independent consumers read the same data and can replay it. Kafka is also the base of most streaming platforms, so it is the safer choice today.

Q8. What are typical use cases for Flume?

Typical cases are collecting web server and application logs into HDFS, moving syslog or network event data into a central store, buffering events through a message queue into Hadoop, and feeding Kafka topics from log files. In each case Flume is the collection layer, and not the processing layer.

Sources, Channels, Sinks and Interceptors Interview Questions for Intermediate

Interviews go through the components one by one. Learn the types and the main property of each.

Q9. What are the common Flume sources?

Common sources are Avro, for events from another Flume agent, Spooling Directory, for completed files placed in a folder, Taildir, which follows files and remembers its position, Kafka, Syslog, HTTP, Netcat and Exec. Taildir is the recommended source for tailing log files, and the Exec source is the one to avoid in production.

Q10. Why is the Exec source not recommended?

The Exec source runs a command, usually tail, and turns its output into events. If the agent or the command dies, the events written meanwhile are lost, because there is no position tracking and no way to confirm what was read. The Taildir source records its position, so it resumes where it stopped.

Q11. What are the common Flume channels?

The memory channel is fast but loses events if the agent stops. The file channel writes events to disk with a checkpoint, so it is durable but slower. The Kafka channel stores events in a Kafka topic, which gives durability and sharing between agents. A JDBC channel exists but is rarely used.

Q12. What are the common Flume sinks?

Common sinks are HDFS, Hive, HBase, Kafka, Avro, which forwards to another agent, Logger, for testing, File Roll and Elasticsearch. The HDFS sink is the best known and writes events as files in HDFS, rolling them by time, size or event count.

Q13. How does the HDFS sink decide when to roll a file, and what is the risk?

The sink closes the current file and starts another when a time, size or event count limit is reached, controlled by hdfs.rollInterval, hdfs.rollSize and hdfs.rollCount. The defaults are low, 30 seconds, 1024 bytes and 10 events, so a busy flow can create a huge number of tiny files. Set sensible values and disable the ones you do not need.

events_per_second = 50
roll_count = 10            # the default hdfs.rollCount

files_per_day = events_per_second * 86400 / roll_count
print("files per day with the default roll count:", int(files_per_day))

files per day with the default roll count: 432000

At 50 events a second, the default rollCount of 10 would make about 432,000 files a day. Most production flows set rollCount to 0 and roll by size or time instead.

Q14. What are interceptors?

Interceptors are plug in components attached to a source that can inspect, change or drop events as they enter a channel. Built in examples add a timestamp, host or static header, extract values with a regular expression or filter events by pattern. They are chained in order, and are often used to set a header that a selector or the HDFS path uses.

Q15. What are channel selectors?

A channel selector decides which channels receive an event from a source that has more than one channel. The replicating selector, which is the default, copies every event to all channels. The multiplexing selector routes an event to a channel based on the value of a header, so, for example, error logs go one way and access logs another.

Q16. What are sink processors and sink groups?

A sink group combines several sinks, and a sink processor decides how they share the work. The default processor uses one sink. The load balancing processor spreads events across the sinks, and the failover processor sends events to the highest priority sink and switches to the next if it fails. They add availability and throughput.

Q17. What is fan in, fan out and multi hop flow?

Fan in means many agents send events to one collector agent. Fan out means one source sends events to several channels and destinations. Multi hop means events pass through more than one agent in a chain. A common design has agents on each server forwarding to a few collector agents, which then write to HDFS.

Reliability and Transactions Interview Questions Experienced Professionals

Reliability questions are the most important for experienced roles. Be clear about what is and is not guaranteed.

Q18. How does Flume guarantee delivery?

Flume uses channel transactions. A source writes events to a channel in a transaction that commits only when they are stored. A sink takes events from the channel in a transaction, and the events are removed only after the sink has delivered them, so a failure causes a retry. This gives at least once delivery, so duplicates are possible.

What the interviewer is listening for: You say at least once, mention that duplicates are possible, and explain that the channel is the point of durability.

Q19. Can Flume lose data, and when?

Yes. A memory channel loses events when the agent crashes. The Exec source can lose events as described. A full channel makes sources fail to add events, so a source that cannot retry may drop data. Use the file or Kafka channel and reliable sources such as Taildir to reduce the risk.

Q20. What are channel capacity and transaction capacity?

Capacity is the maximum number of events a channel can hold, and transactionCapacity is the maximum number of events it accepts or gives in one transaction. The batch size of a source or sink must not exceed the transaction capacity. If the sink is slower than the source, the channel fills up, and the source starts failing.

Q21. How do you handle backpressure in Flume?

When a sink is slow, the channel fills, and the source cannot add events. Add sinks with a load balancing processor, raise the batch size, use a larger or file channel as a buffer, and fix the cause at the destination. Monitor channel fill percentage, since it is the first sign of trouble.

Q22. How do you avoid duplicates from Flume?

Duplicates come from at least once delivery after retries. Add a unique event ID in an interceptor and deduplicate downstream, or write to a destination that can upsert on that ID. Accept that Flume cannot give exactly once by itself.

Q23. How do you make a Flume topology highly available?

Use two or more collector agents, and configure the upstream agents with a failover or load balancing sink group, so events keep moving if one collector fails. Use durable channels such as the file or Kafka channel. Run agents under a supervisor that restarts them, and monitor them.

Configuration and Operations Interview Questions for Intermediate

You will be asked to read a configuration and to say what it does.

Q24. How is a Flume agent configured?

A properties file names the agent, then lists its sources, channels and sinks, then sets the properties of each and binds sources and sinks to channels. The agent name used at start up must match the prefix in the file. This example tails log files, buffers them in a file channel and writes to HDFS.

a1.sources  = r1
a1.channels = c1
a1.sinks    = k1

a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /var/log/app/.*log
a1.sources.r1.positionFile = /var/flume/taildir_position.json
a1.sources.r1.channels = c1

a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /var/flume/checkpoint
a1.channels.c1.dataDirs = /var/flume/data

a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://nn:8020/logs/%Y/%m/%d
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.rollInterval = 300
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.channel = c1

Q25. How do you start a Flume agent?

Use the flume ng command with the agent option, naming the agent, the configuration directory and the configuration file. The name must match the prefix in the file. Add a logging property to see output on the console while testing.

flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console

Q26. How do you monitor Flume?

Flume exposes metrics through JMX and, if you start it with the monitoring option, as JSON over HTTP. Watch the channel fill percentage, the number of events received, accepted and delivered, and the sink failures. Feed them to a monitoring system, and alert when the channel stays nearly full.

flume-ng agent --conf conf --conf-file example.conf --name a1 \
  -Dflume.monitoring.type=http -Dflume.monitoring.port=34545

Q27. How do you partition HDFS output by date using Flume?

Put time escape sequences in hdfs.path, such as year, month and day codes. Flume fills them from the timestamp header of each event. If events have no timestamp, set useLocalTimeStamp to true, or add a timestamp interceptor. Using the event time, not the arrival time, is better when events arrive late.

Q28. How do you send events to several destinations at once?

Define one source with several channels and sinks, and use the replicating channel selector, so each event is copied to every channel. Each channel then feeds its own sink, such as one HDFS sink and one Kafka sink. Use the multiplexing selector instead if each event should go to only one destination.

Q29. How do you test a Flume configuration safely?

Use a Netcat or sequence generator source with a Logger sink and a memory channel, start the agent with console logging and send a few test events. Then swap in the real components one at a time. Test failure cases by stopping the destination and watching the channel fill.

Flume Against Modern Tools for All Levels

A senior interviewer may ask what you would use instead. Have a clear and fair answer.

Q30. What would you use instead of Flume for a new log pipeline?

A common design is lightweight agents such as Fluent Bit, Filebeat or Vector on each server, sending to Kafka as a durable buffer, then Kafka Connect or Spark Structured Streaming writing to object storage or HDFS in Parquet. This is actively maintained, supports replay and scales with Kafka. For simple flows, a managed cloud ingestion service may be enough.

What the interviewer is listening for: You give a specific and current stack and explain the role of Kafka as a buffer.

Q31. How does Flume compare with Apache NiFi?

NiFi is a flow management tool with a web interface for designing, running and monitoring data flows, with back pressure, data provenance and many processors, and it is actively maintained. Flume is configuration file based and lighter. NiFi suits teams that want a visual tool and lineage, and Flume suits fixed log shipping into Hadoop.

Q32. How do Logstash and Beats compare with Flume?

Beats, such as Filebeat, are lightweight shippers that read files and send them on, and Logstash is a heavier processing pipeline with many filters. They are designed around the Elastic Stack, though they can output to Kafka and other systems. Flume is designed around Hadoop and HDFS as the main destination.

Q33. How do you migrate a Flume pipeline to Kafka?

Keep the Flume agents running, and add a Kafka sink or channel so events also flow into a topic. Build the new consumer, such as Kafka Connect to HDFS, and compare its output with the old output. When they match, retire the Flume to HDFS path and replace the agents with a modern shipper over time.

Q34. Does Flume still make sense to learn in 2026?

Learn it if you will support an existing Hadoop pipeline or if a job description names it. The concepts of sources, buffers and sinks, transactional hand off, fan in and fan out, and back pressure transfer to every ingestion tool. For general data engineering, learn Kafka first and treat Flume as a legacy topic.

Flume Scenario Interview Questions for Experienced Professionals

State the cause, the fix and the prevention.

Q35. Your HDFS directory has millions of tiny files created by Flume. What do you do?

The HDFS sink is rolling too often. Raise hdfs.rollInterval, set hdfs.rollSize near the block size, and set rollCount to zero. Compact the existing files with a scheduled job into larger files or Parquet, and add monitoring on file counts. Reduce the number of sinks writing to the same path.

Q36. The Flume channel is almost full and events are being rejected. What do you do?

The sink is slower than the source. Check the destination for problems, such as HDFS slowness or a full disk. Increase sink batch size, add more sinks in a load balancing group, and use a larger file channel as a buffer. If the load has permanently grown, add collector agents.

Q37. After an agent crash, some log lines were missing. How do you find the cause?

Check the source and channel types. A memory channel or Exec source explains lost events, because both lose data on a crash. Switch to the Taildir source and a file or Kafka channel. Check the log and checkpoint directories, and compare event counts from source to destination to confirm that the gap is closed.

Q38. You see duplicate log lines in HDFS. Why?

Flume delivers at least once, so a retry after a failed sink transaction can write events again. A Taildir position file that was reset can also re read old files. Add a unique ID and deduplicate downstream, protect the position file and avoid restarting agents in a way that loses it.

Q39. Your company asks whether to replace a working Flume pipeline. What do you advise?

If it works and is stable, there is no urgent reason, but the project is dormant, so plan an exit. Keep it monitored, document it, and fix risky parts such as a memory channel. Build any new pipeline on Kafka. Replace Flume when you touch the pipeline for other reasons, or when a security or Java upgrade blocks it.

What the interviewer is listening for: You weigh risk and cost, and you do not recommend replacing a stable system without a reason.

Common questions about Flume interviews

Is Flume still used in 2026?

It still runs in many existing Hadoop log pipelines, but the project has had little activity since late 2024 and new projects rarely choose it. Most teams building new ingestion use Kafka with Kafka Connect or a lightweight log shipper. Learn Flume mainly to support or migrate legacy systems.

What topics should I prepare for a Flume interview?

Prepare the agent architecture, the event structure, the main source, channel and sink types, and the difference between memory and file channels. Then learn delivery guarantees, the HDFS sink roll settings, interceptors, selectors and a sample configuration. Be ready to compare Flume with Kafka.

Which Flume version is current?

The latest release is 1.11.0, and the project has been mostly dormant since late 2024. Check the Apache Flume site for the latest status before an interview or an upgrade, since the position may change.

Do I need Java to use Flume?

Flume is written in Java and needs a supported Java runtime, but most work is writing a properties file. Java is needed only to write a custom source, sink or interceptor.

How can I practise Flume at home?

Install Flume on a laptop, run a Netcat source with a Logger sink to see events move, then tail a log file with Taildir into a file channel and an HDFS or local file sink. Stop the destination on purpose and watch the channel fill.

A four week plan to prepare

Use this plan if you have about a month before your interview.

Practise with a trainer

If you want to practise these questions with a trainer and work on hands on projects, you can book a free demo class for the Prwatech Big Data course and Hadoop course. It covers Hadoop, Kafka, Spark, data ingestion and hands on projects.

Contact the learning team for current batch dates and fees.

Written by Mithun Ray, data engineering trainer at Prwatech. Reviewed by Avishekh Verma. Last updated October,2026.