[Message Queue Case Study] Kafka Whitepaper Summary and Architecture

This is a summary of the Kafka white paper, which is available at this link.

1. Abstract

• Kafka, a distributed messaging system that is developed for collecting and delivering high volumes of log data with low latency.
• Kafka has superior performance when compared to two popular messaging systems.

2. Introduction

• A large amount of log data is generated at any sizable internet company which includes user activity, operational metrics.

  1. User activity
     • It contains events corresponding to logins, page views, clicks, comments, and search queries.
  2. Operational metrics
     • It contains service call stack, call latency, errors, and system metrics
  3. system metrics include CPU, memory, network, or disk utilization on each machine.
  4. Activity data

1. search relevance
2. recommendations
3. ad targeting and reporting
4. security

• Kafka provides an API similar to a messaging system and allows applications to consume log events in real-time.