Kafka Topic

Topic- a particular stream of data:

Kafka topics are a particular stream of data with in your kafka cluster.

Like a table in a database:

We can say that a topic is similar to a table in a database but without all the constraints and we can't able to query with a kafka topic.

You can have as many topics as you want:

You can have as many topics as you want in your kafka cluster.

A Topic is identified by its name:

The way to identify a topic in kafka cluster is by its name.

Any kind of message format:

Kafka topic support any kind of message format, you can send for example JSON, Avro, Text file, binary whatever you want.

The sequence of message is called a data stream:

The sequence of message in a topic is called data stream and this is why kafka is called data streaming platform, because it makes data stream through topic.

Partitions and Offsets

Topics are split in partitions :

You can divide the topic into partition. A topic can be made up of, for example 100 partitions

Messages within each partition are ordered:

Messages send to kafka topic are going to endup in these partitions and messages with in each partition are going to be ordered.

Suppose, my first message are going into partition 0 and have ID 0 and then 1,2,....so on, and as it keep writing messages into my partition this ID is going to increase.

This is the same case when I goto write data into partition 1 into my kafka topic, these ID will keep on increasing and so on.

Each message within a partition gets an incremental ID, called offset:

The messages in these partitions, where they are written they are getting an ID, thats increment from 0 to 1 and so on..., and these ID in kafka partition is called an offset.

Kafka topics are immutable- Once data is written to a partition, it can not be changed:

So, we can not delete data in kafka and also we can not update data in kafka. We have to keep writing to the partition.

Dta is kept only for a limited time (default is one week-configurable):

It means your data after one week will disapear.

offsets are not re-used if previous message have been deleted:

It keeps on increasing incrementally one by one, as you send the messages in to your kafka topic

Order is guaranteed only with in a partition (not across partitions):

The messages with in each partition they have offset increasing, so that they are in order.

But across partition we have no controll.

Data is assigned randomly to a partition unless a key is provided:

It means if you are sending a data in key value pair. So, for same key your data will assign to a same partition.

You can have as many partitions per topic as you want.