Using Apache Kafka for Asynchronous Communication in Microservices

  • In this post, we’ll look at how to set up an Apache Kafka instance, create a user service to publish data to topics, and build a notification service to consume data from those topics.
  • One of the required configuration options for the Kafka service is , which tells Kafka where to find the Zookeeper instance.
  • To publish data to a Kafka topic, we are going to create a user service that provides two endpoints: – – We use the NPM package to create a producer that connects to Kafka from our node app: – – We create a new promise object that resolves to a…
  • This is used in our function, which publishes data to a Kafka topic partition: – – To consume data from our Kafka topic, we are going to create a notification service that listens for data coming from our topics and sends an email with either a verification code or success…
  • Our two-factor authentication app demonstrates the communication pattern between only two microservices using Apache Kafka (there are other systems like RabbitMQ, ZeroMQ), but by decoupling communication between those services, we add flexibility for the future.

if you’re considering microservices, you have to give serious thought to how the different services will communicate. In this post, we’ll look at how to set up an Apache Kafka instance, create a user service to publish data to topics, and build a notification service to consume data from those topics.

While microservice architecture might not be a silver bullet for all systems, it definitely has its advantages, especially when building a complex system with a lot of different components. Of course, if you’re considering microservices, you have to give serious thought to how the different services will communicate.

In this post, we’ll look at how to set up an Apache Kafka instance, create a user service to publish data to topics, and build a notification service to consume data from those topics. Specifically, we’re going to build a two-factor authentication app where a user will register, receive a mail with a verification code, and complete their registration using the code. The source code can be found here.

Kafka is a distributed streaming platform created by LinkedIn in 2011 to handle high throughput, low latency transmission, and processing of streams of records in real time. It’s three major capabilities make it ideal for this use case:

Before starting this tutorial, the following will be required:

We will be using the Wurstmeister Kafka Docker image. Note that Kafka uses Zookeeper for coordination between different Kafka nodes.

A similiar to the one below is used to pull the images for Kafka and Zookeeper. One of the required configuration options for the Kafka service is , which tells Kafka where to find the Zookeeper instance.

Using Apache Kafka for Asynchronous Communication in Microservices