# Kafka Made Simple: A Hands-On Quickstart with Docker and Spring Boot

Apache Kafka is a distributed, durable, real-time event streaming platform. It goes beyond a message queue by providing scalability, persistence, and stream processing capabilities.

In this guide, we’ll quickly spin up Kafka with Docker, explore it with CLI tools, and integrate it into a Spring Boot application.

---

## 1\. What is Kafka?

Apache Kafka is a **distributed, durable, real-time event streaming platform**.  
It was originally developed at LinkedIn and is now part of the Apache Software Foundation.  
Kafka is designed for **high-throughput, low-latency data pipelines, streaming analytics, and event-driven applications**.

### What is an Event?

An **event** is simply a **record of something that happened** in the system.  
Each event usually includes:

* **Key** → identifier (e.g., user ID, order ID).
    
* **Value** → the payload (e.g., “order created with total = $50”).
    
* **Timestamp** → when the event occurred.
    

Example event:

```json
{
  "key": "order-123",
  "value": { "customer": "Alice", "total": 50 },
  "timestamp": "2025-09-19T10:15:00Z"
}
```

### What is an Event Streaming Platform?

An **event streaming platform** is a system designed to handle continuous flows of data — or *events* — in real time.  
Instead of working in batches (processing data after the fact), it allows applications to **react as events happen**.

---

## 2\. What Kafka Can Do

Kafka is more than a message queue—it's a real-time event backbone for modern systems.

### Messaging Like a Message Queue

Kafka decouples producers and consumers, enabling asynchronous communication between services.

**Example**: A banking system publishes transaction events to Kafka. Fraud detection, ledger updates, and notification services consume these events independently.

### Event Streaming

Kafka streams data in real time, allowing systems to react instantly.

**Example**: An insurance platform streams claim events to trigger automated validation, underwriting checks, and customer updates in real time.

### Data Integration

Kafka Connect bridges Kafka with databases, cloud storage, and analytics platforms.

**Example**: A semiconductor company streams sensor data from manufacturing equipment into a data lake for predictive maintenance and yield optimization.

### Log Aggregation

Kafka centralizes logs from multiple services for monitoring and analysis.

**Example**: An industrial automation system sends logs from PLCs and controllers to Kafka, where they’re consumed by a monitoring dashboard for anomaly detection.

### Replayable History

Kafka retains events for reprocessing or backfilling.

**Example**: An insurance company replays past policy events to train a model that predicts claim risk or customer churn. This avoids relying on static snapshots and gives the model a dynamic, time-aware view of behavior.

### Scalable Microservices Communication

Kafka handles high-throughput messaging across distributed services.

**Example**: A financial institution uses Kafka to coordinate customer onboarding, KYC checks, and account provisioning across multiple microservices.

---

## 3\. Architecture

Apache Kafka’s architecture is built for **high throughput, fault tolerance, and horizontal scalability**. At its core, Kafka relies on a **log-based storage model** and a **distributed broker cluster**.

**Core Components**

* **Producer** → Publishes records (events/messages) to topics. Can be idempotent or transactional.
    
* **Topic** → Logical category/feed for messages. Divided into **partitions** for parallelism.
    
* **Partition** → Ordered, immutable commit log. Records have sequential **offsets**.
    
* **Broker** → A Kafka server that stores partitions. Clusters have multiple brokers.
    
* **Consumer** → Subscribes to topics and processes messages. Part of a **consumer group** for scaling.
    
* **Controller** → Special broker role that manages metadata, leader election, and partition assignment.
    
* **Replication** → Each partition has one **leader** and multiple **followers** in the **ISR** (in-sync replicas).
    

 **Data Flow**

1. **Producers** send records to brokers.
    
2. Records are appended to the **leader partition log**.
    
3. Followers replicate the leader’s log for durability.
    
4. **Consumers** fetch records from leaders, tracking their **offsets**.
    

 **Architecture Diagram**

```plaintext
               +-----------------+
               |    Producers    |
               +-----------------+
                   |    |    |
                   v    v    v
            +------------------------+
            |     Kafka Cluster      |
            |  +---------+           |
            |  | Broker 1|  <--------------- Partition 0 Leader
            |  +---------+           |
            |  | Broker 2|  <--------------- Partition 0 Follower
            |  +---------+           |
            |  | Broker 3|  <--------------- Partition 1 Leader
            |  +---------+           |
            +------------------------+
                   |    |    |
                   v    v    v
              +-------------------+
              |  Consumer Group   |
              |-------------------|
              | Consumer A → P0   |
              | Consumer B → P1   |
              +-------------------+
```

---

## 4\. QuickStart with Docker

This configuration sets up a single-node Kafka broker using the KRaft. It’s ideal for development, testing scenarios

```yaml
name: kafka
services:
  kafka:
    image: apache/kafka:4.1.0
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: BROKER://:9092,CONTROLLER://:9093
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: BROKER://localhost:9092
      KAFKA_CLUSTER_ID: "kafka-1"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka_data:/var/lib/kafka/data
    ports:
      - "9092:9092"
volumes:
  kafka_data:
```

## How to Run

Start the Kafka container using:

```bash
docker compose up
```

Kafka will be available at localhost:9092 for producers and consumers, and internally at localhost:9093 for controller communication.

---

## 5\. Kafka CLI

Before running Kafka commands, log into the Kafka container:

```bash
docker container exec -it localhost bash
```

### Create Topic

Create a topic named quickstart with one partition and a replication factor of 1:

```bash
/opt/kafka/bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --replication-factor 1 \
  --partitions 1 \
  --topic quickstart
```

### List Topic

Check all existing topics:

```bash
/opt/kafka/bin/kafka-topics.sh --list \
  --bootstrap-server localhost:9092
```

### Consume Message

Read messages from the order topic starting from the beginning:

```bash
/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic quickstart \
  --from-beginning
```

### Send Message

You can send messages to the quickstart topic using either direct input or a file.

### Option A: Send a single message

```bash
echo 'This is Event 1' | \
/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic quickstart
```

### Option B: Send multiple messages from a file

```java
echo 'This is Event 2' > messages.txt
echo 'This is Event 3' >> messages.txt
cat messages.txt | \
/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic quickstart
```

---

## 5\. Spring Boot Integration

This configuration enables seamless integration between a Spring Boot application and an Apache Kafka broker. It defines both producer and consumer settings for message serialization, deserialization, and connection behavior.

### pom.xml

```xml
<!-- spring-web -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <version>3.4.9</version>
</dependency>
<!-- kafka -->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>3.3.9</version>
</dependency>
<!-- Lombok(optional) -->
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.30</version>
    <optional>true</optional>
</dependency>
```

### applicaiton.yml

```yaml
spring:
  kafka:
    bootstrap-servers: localhost:9092
    template:
      default-topic: orders
    consumer:
      group-id: quickstart-group
      auto-offset-reset: latest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "dev.aratax.messaging.kafka.model"
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
```

### Topic Setup

```java
@Bean
public NewTopic defaultTopic() {
    return new NewTopic("orders", 1, (short) 1);
}
```

### Event Model

```java
public class OrderEvent {
    private String id;
    private Status status;
    private BigDecimal totalAmount;
    private Instant createdAt = Instant.now();
    private String createdBy;

    public enum Status {
        IN_PROGRESS,
        COMPLETED,
        CANCELLED
    }
}
```

### Producer Example

```java
@RestController
@RequestMapping("/api")
@RequiredArgsConstructor
public class OrderEventController {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    @PostMapping("/orders")
    public String create(@RequestBody OrderEvent event) {
        event.setId(UUID.randomUUID().toString());
        event.setCreatedAt(Instant.now());
        kafkaTemplate.sendDefault(event.getId(), event);
        return "Order sent to Kafka";
    }
}
```

### Consumer Example

```java
@Component
public class OrderEventsListener {

    @KafkaListener(topics = "orders")
    public void handle(OrderEvent event) {
        System.out.println("Received order: " + event);
    }
}
```

---

## 6\. Demo Project

I built a demo project using Spring Boot and Kafka to demonstrate basic producer/consumer functionality. Check it out on GitHub: [springboot-kafka-quickstart](https://github.com/arata-x/springboot-kafka-quickstart)

---

## 7\. Key Takeaways

* Kafka is more than a message queue—it's a scalable, durable event streaming platform.
    
* Events are central to Kafka’s architecture, enabling real-time data flow across systems.
    
* Docker makes setup easy, allowing you to spin up Kafka locally for development and testing.
    
* Kafka CLI tools help you explore topics, produce messages, and consume events interactively.
    
* Spring Boot integration simplifies Kafka usage with built-in support for producers and consumers.
    
* Real-world use cases span industries like banking, insurance, semiconductor, and automation.
    

---

## 8\. Conclusion

Apache Kafka empowers developers to build reactive, event-driven systems with ease. Whether you're streaming financial transactions, processing insurance claims, or monitoring factory equipment, Kafka provides the backbone for scalable, real-time communication.

With Docker and Spring Boot, you can get started in minutes—no complex setup required. This quickstart gives you everything you need to explore Kafka hands-on and begin building production-grade event pipelines.

Ready to go deeper? Try explore its design/implementation, stream processing, or Kafka Connect integrations next.