0%

Kafka Study: Part II, Integration with Spring Boot

March 22, 2026

Java

Kafka

Message-Broker

Streaming

Contents

1. Repository

1.2.

How to use this Repository?

  • cd into kafka-cluster/ and run docker-compose up, a kafka-cluster of 3 instances will be launched, with localhost:9092 as the entrypoint.

  • Both consumer/ and producer/ are spring applications

  • Launch the spring application in consumer/

  • Launch the spring application in producer/, this will launch a backend server at port: 8081

  • Go to localhost:8081, a swagger document has been launched to create message:

  • Send the POST request to create a message in producer/, and receive the message from consumer/ application

2. Create Topics in Spring Boot via Spring Configuration

package com.machingclee.kafka.common.type;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.config.TopicBuilder;
import org.springframework.context.annotation.Configuration;

@Configuration
class KafkaTopicConfig {
    @Bean
    public NewTopic topic1() {
        return  TopicBuilder.name("my-topic")
                .partitions(3)
                .replicas(3)
                .build();
    }

    @Bean
    public NewTopic topic2() {
        return  TopicBuilder.name("topic-created-by-spring")
                .partitions(3)
                .replicas(3)
                .build();
    }
}

3. Producer

3.1.

KafkaController

We have made a simple controller to create a message into the Kafka topic via swagger document:

@RestController
@RequestMapping("/kafka")
public class KafkaController {
    private final KafkaProducerService kafkaProducerService;

    public KafkaController(KafkaProducerService kafkaProducerService) {
        this.kafkaProducerService = kafkaProducerService;
    }

    @PostMapping("/add-course")
    public ResponseEntity<String> addCourse (@RequestBody Course course) {
        // send course to kafka service

        String response = kafkaProducerService.sendMessage(course);
        return ResponseEntity.ok(response);
    }
}

3.2.

KafkaProducerService

1@Service
2public class KafkaProducerService {
3    private final KafkaTemplate<String, Course> kafkaTemplate;
4
5    public KafkaProducerService(KafkaTemplate<String, Course> kafkaTemplate) {
6        this.kafkaTemplate = kafkaTemplate;
7    }
8
9    public String sendMessage (Course course) {
10        this.kafkaTemplate.send("my-topic", "course", course);
11        return "Course message sent to Kafka service";
12    }
13}

In line 10 we have used the following method overloading:

(topic, groupId, messageObject) -> SendResult

4. Consumer

4.1.

The Course Record

package com.machingclee.kafka.common.type;

public record Course(
        String courseId,
        String title,
        String trainer,
        double price
) {}

4.2.

KafkaConsumerService

@Service
public class KafkaConsumerService {
    private final KafkaTemplate<String, Course> kafkaTemplate;
    private String message;
    public KafkaConsumerService(KafkaTemplate<String, Course> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    @KafkaListener(topics = "my-topic", groupId = "my-group")
    public String getMessage (Course course) {
        System.out.println("Received course: " + course);
        return "Course message received from Kafka service";
    }
}

4.3.

Setg GroupID in application.yml

Ideally each instance of spring application should have only one group id.

Instead of hardcoding the group ID in @KafkaListener, it can be externalised to application.yml and referenced via a property placeholder:

spring:
  kafka:
    consumer:
      group-id: "course-group"
@KafkaListener(topics = "my-topic"")
public void listen(Course course) { ... }

Spring uses spring.kafka.consumer.group-id as the default when no explicit groupId is set on @KafkaListener.

4.4.

Concurrency

4.4.1.

Parallel Consumption Within One Instance

By default a @KafkaListener spawns one consumer thread, processing messages from its assigned partition sequentially. Setting concurrency spawns multiple consumer threads inside the same JVM — each thread acts as an independent consumer within the group and is assigned its own partition:

@KafkaListener(
    topics = "my-topic", 
    groupId = "my-group", 
    concurrency = "3"
)
public void listen(Course course) { ... }
Spring starts 3 consumer threads, all in group "my-group"
                ↓
Group coordinator assigns partitions:
  thread-1 → partition 0
  thread-2 → partition 1
  thread-3 → partition 2

thread-1 processes: msg-a, msg-b ...   ─┐
thread-2 processes: msg-d, msg-e ...   ─┼─ in parallel, same machine
thread-3 processes: msg-g, msg-h ...   ─┘

From Kafka's perspective these are 3 separate consumers — it doesn't know or care they share a JVM. The partition assignment works exactly the same as running 3 separate application instances.

The partition cap still applies: concurrency threads beyond the partition count sit idle. Setting concurrency=3 for a 1-partition topic gives us 1 active thread and 2 idle ones.

There is no shared thread pool to configure — Spring Kafka creates exactly concurrency dedicated threads per listener container and keeps them alive for the lifetime of the application. If we have 2 @KafkaListener methods each with concurrency=3, we get 6 total consumer threads.

4.4.2.

What if multiple machines also use concurrency="3"?

The same partition cap applies across the entire group. With 3 machines × concurrency=3 = 9 consumer threads total, all competing for 3 partitions:

Topic: 3 partitions

Machine A:  thread-1 → partition 0  ✓ active
            thread-2 → partition 1  ✓ active
            thread-3 → partition 2  ✓ active

Machine B:  thread-4 → (nothing)    ✗ idle
            thread-5 → (nothing)    ✗ idle
            thread-6 → (nothing)    ✗ idle

Machine C:  thread-7 → (nothing)    ✗ idle
            thread-8 → (nothing)    ✗ idle
            thread-9 → (nothing)    ✗ idle

We get zero extra throughput over a single machine with concurrency=3. To actually utilise all 9 threads we need 9 partitions:

9 partitions, 3 machines × concurrency=3:

Machine A: partition 0, 1, 2
Machine B: partition 3, 4, 5
Machine C: partition 6, 7, 8

This is why partition count must be planned ahead — it is the hard ceiling on total parallelism across the entire consumer group, regardless of how many machines or threads we add.

4.5.

Virtual Threads (Spring Boot 3.2+ / Java 21+)

Enabling virtual threads is a one-line change in application.yml:

spring:
  threads:
    virtual:
      enabled: true

Spring Boot automatically wires a VirtualThreadTaskExecutor into the Kafka listener container factory. Each consumer thread becomes a virtual thread — extremely lightweight compared to OS threads, so the cost of having many of them is negligible.

The two settings are orthogonal:

SettingWhat it controls
concurrencyHow many consumer threads exist (still capped by partition count)
spring.threads.virtual.enabledWhat kind of thread they are (OS thread vs. virtual thread)

Virtual threads matter most when the listener does blocking I/O (DB calls, HTTP calls). For fast, CPU-bound processing the difference is minimal.

4.6.

Result on Message Received