Topic Replication Factor

Topics should have a replication factor > 1:

In production environment kafka have replication factor more than 1 usually between 2 and 3. Replication factor means the number of duplicate copy you want of your each partition.

This way if a broker is down another broker can serve the data:

If a kafka server is down or stopped for maintenance, the another kafka broker still has a copy of the data to serve and receive.

Example: Topic-A with 2 partitions and replication factor of 2:

In block diagram, we have a Topic-A it has two partitions and a replication factor of two.

So we have three kafka brokers which contain Topic-A original partition and replication partition.

Example: we lose broker 102

Suppose, if we lose broker 102 the we can see we have broker 101 and 103 still up and they can still serve the data.

Result: Broker 101 and 103 can still serve the data

So, partition 0 and partition 1 are still available with in our cluster and this is why we have a replication factor.

Concept of leader for a partition

At any time only One broker can be a leader for a given partition:

At any time only one broker can be a leader for a given partition.

Producers can only send data to the broker that is leader of a partition:

Producer can only send data to thebroker that is the leader of a partition.

In block diagram, we can see a star is addedon the leader of each partition, and we can see Broker 101 has the leader of partition 0 and Broker 102 has the leader of partition 1, but Broker 102 has the replica of partition 0 and Broker 103 has a replica of partition 1.

The other brokers will replicate the data, therefor each partition has one leader and multiple ISR(in-sync-replica).

Default producer & consumer brhaviour with leaders

Kafka producers can only write to the leader broker for a partition:

By default, producers are going to only write into the leader broker for a partition. So, if the producer knows it wants to send data into partition 0 then the producer will always choose the leader of the partition to send data.

Kafka consumers by default will read from the leader broker for a partition:

Kafka consumer are going to read by default only from the leader of a partition.

Kafka consumers replica fetching (Kafka v2.4+)

Since Kafka 2.4 it is possible to configure consumers to read from the closest replica:

It means in Kafka 2.4 and later version one new feature was introduced, which allows consumer to read from the closest replica.

In block diagram, we have the Broker 101, which is the leader of partitiop 0, receiving the data from the producer is going to replicate data into the ISR partition 0 of Broker 102, then it is possible for our consumer to read from the replica itself.

This may help improve latency and also decrease network cost if using the cloud:

It means, this may help to improve latency because may be the consumer is really close to Broker 102 and also may be it's going to help decrease network cost if using the cloud, because if things are in the same data center then you have little to no cost.