LaVOZs

The World’s Largest Online Community for Developers

'; java - Spring Kafka: Read from two different topics in order - LavOzs.Com

Is it possible with Spring Kafka to read from two different topics in different consumers in a guaranteed order? For example, topic A stores information that is pertinent to determine how to handle the data that is stored in topic B. Topic A is read into memory and referenced, but needs to be completely populated before reading data in from topic B. Below is an example of what my current setup looks like...

@Service
public class TopicA {
  @KafkaListener(topics = "topicA")
  public void consume(ConsumerRecord<String, byte[]> record) {
    // ... some code here to populate an in-memory data structure
  }
}
@Service
public class TopicB {
  @KafkaListener(topics = "topicB")
  public void consume(ConsumerRecord<String, byte[]> record) {
    // ... some code here that depends on topic A having populated the in-memory data structure
  }
}

So far I've been leaning towards creating a Spring startup process (using @PostConstruct) that initializes the data structure by reading from topic A first, but have not been able to get that working. Does anyone have any suggestions? Thanks in advance!

@KafkaListener(id = "bConsumer" topics = "topicB", autoStartup = "false")

Then autowire the KafkaListenerEndpointRegistry bean and registry.getListenerContainer("bConsumer").start(); when you are ready.

Related
What's the difference between @Component, @Repository & @Service annotations in Spring?
Data Modeling with Kafka? Topics and Partitions
Is it possible to disable caching for some kafka topics?
Database connection lost when reading from Kafka
How to skip corrupt (non-serializable) messages in Spring Kafka Consumer?
How to read all the records in a Kafka topic
How to re-send (read) an old kafka message from given topic and partition at specific offset using spring-kafka?
Migrate apache kafka polling to spring kafka
How to configure multiple kafka consumer in application.yml file
Consuming from kafka topic using spring only on receiving certain events