Apache Kafka

Apache Kafka is a broker or mediator between the message sender and receiver. We can call source system as a producer and target system as a consumer in Kafka terminology.

Kafka acts as a central messaging hub, allowing applications (producers) to send data to kafka topics without directly communicating with consumers.

This decoupling enables applications to focus on their core functionality without being tightly coupled to other application.

It is designed to handle large volumes of real-time data streams, meaning it can process data as it is being produced, making it suitable for applications that require immediate responsees to event.

Distributed:

Apache Kafka ia a distributed messaging system, and its distributed nature allows for scalability which is crucial for handling the demands of modern data pipelines and streaming applications.

Resilient Architecture:

Kafka achives resiliancy through a distributed and replicated architecture.

Fault Tolerance:

Kafka replicates data accross multiple brokers incuring that data remains available even if some brokers fails.

Horizontal Scalability:

Kafka is built to handle massive amount of data and can be scaled horizontally by adding more brokers to the cluster, which allows for parallel processing and efficient data distribution.

It can scale to millions of messages per second.

High Performance:

It have really low latency sometime measure less than 10 miliseconds, this makes kafka a real time system

Used by 2000+ firms:

Kafka have wide adaptability accross the world, over 2000 firms are using kafka publicly and also over 80% of the fortune 100 companies are using kafka.

Apache Kafka : Use Cases

Messaging System:

Kafka can handle a large volume of messages with high throughput, making it suitable for real-time data processing, like twitter and other messaging application

Activity Tracking:

Applications and websites generate activity events like website visit, clicks, page views, searches and form submission. We can track all these activities using kafka

Gather metrics:

Kafka can gather metrics from many different locations and by analysing the incoming metric streams, kafka can be used to detect anomalies or deviations from expected behaviour, and alert can be triggered based on predefined thresholds

Applications logs gathering:

Kafka can be used used to aggregate logs from multiple sources and make them available for analysis

Stream processing:

Kafka is used for real-time processing of continuous streams of data. For eg. Analyzing markrts data feeds, detecting trading anomalies, and making real-time trading decision.