Kafka Brokers

A kafka cluster is composed of multiple brokers(servers):

A kafka cluster can have multiple kafka brokers. Kafka broker is just a server but in kafka they called brokers because they receive and send data.

Each brokers is identified with its ID:

A kafka brokers is identified with an ID, which is an integer.

For eg. In block diagram we are having broker 101, broker 102 and broker 103 in our cluster.

Each broker contains certian topic partitions:

Each brokers is going to contain only certain topic partitions, that means your data is going to be distributed accross all brokers.

After connecting to any broker(called a bootstrap broker), you will be connected to the entire cluster:

After you connect with any kafka broker then the clients will be connected and know how to connect to the entire kafka cluster, because the kafka vlients have a smart mechanism for this.

That means you don't need to know in advance all the brokers in your cluster, you just need to know how to connect to any one broker with in the cluster and then your clients will automatically connect to the rest.

Brokers and Topics

Example of Topic-A with 3 partition and Topic-B with 2 Partitions:

In block diagram we can see, we have a Topic-A with 3 partitions and a Topic-B with 2 partitions and we have 3 kafka brokers 101, 102 and 103. Broker 101 contains Topic-A partition 0 and Topic-B partition 1, Broker 102 contains Topic-A partition 2 and Topic-B partition 0, Broker 103 contains Topic-A partition 1.

Note- data is distributed and Broker 103 does not have any Topic-B data:

In block diagram, we can see that data is distributed and it's normal that Broker 103 does not have any Topic-B data partition, because the two partitions have already been placed on our kafka brokers.

Ths data and your partitions is going to be distributed across all brokers and this is what makes kafka scale and this is actually called horizontal scaling.

Kafka Brokers Discovery

Every kafka brokers is also called a bootstrap server:

Each kafka brokers in your cluster is called a bootstrap server. As we can see in this block diagram we have 5 bootstrap server in our kafka cluster.

You only need to connect to one broker, and the kafka clients will know how to connect to the entire cluster:

Step 1: Kafka client is going to initiate the connection with broker 101.

Step 2: Then if the connection is successful, the broker 101 is going to return the list of all the brokers available in the cluster.

Step 3: The kafka client can able to connect with list of all the brokers available inside the kafka cluster.

Each broker knows about all brokers, topic and partitions metadata:

Each broker have all metadata information of all brokers in the cluster.