Today, we will be exploring how to set up a local Kafka cluster using Docker. This article will be referenced by future tutorials that require a working Kafka cluster. There are a few good tutorials already about how to set up a Kafka cluster locally, but I wasn’t completely satisfied with any of them. This is my attempt at covering the process in hopes that it will help someone else who is interested in both Docker and Kafka to create a usable development environment in just a few minutes.
The following items will need to be installed for this tutorial.
# Verify that Docker is installed
$ docker --version
Docker version 1.12.5, build 7392c3b
# Verify that VirtualBox is installed
$ vboxmanage -v
5.0.10r104061
# Verify that kafkacat is installed
$ which kafkacat
/usr/local/bin/kafkacat
Note: if you would like to use another provider other than VirtualBox (e.g. VMWare Fusion, Digital Ocean, Amazon Web Services, etc), you can forgo the Virtual box installation above, and follow the directions for using Docker Machine with your chosen provider.
The first thing we need to do is create a Docker host using the docker-machine command.
$ docker-machine create --driver virtualbox kafka-cluster
Once the above command has finished executing, you can verify that the machine was created by running:
$ docker-machine ls
You should see output similar to the following:
# output
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
kafka-cluster - virtualbox Running tcp://192.168.99.100:2376 v1.12.5
The IP address in the URL column above will be needed for later steps. This will be the IP of our broker and Zookeeper node. Another way to retrieve this value is to run the docker-machine ip command. For example:
$ docker-machine ip kafka-cluster
# 192.168.99.100
Now, in order to set the kafka-cluster host as the active environment, we need to use the docker-machine env command. This basically tells Docker which host to run the containers on when we bring them online in the steps below.
$ eval $(docker-machine env kafka-cluster)
Now that we have created a dedicated Docker host for our Kafka cluster, lets run a broker and Zookeeper node locally using the popular wurstmeister/kafka-docker image. First, clone the project using the following command:
$ git clone git@github.com:wurstmeister/kafka-docker.git
$ cd kafka-docker
Now, within the kafka-docker directory, add the following configuration to a file named docker-compose-kafka-cluster.yml.
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_CREATE_TOPICS: "test:10:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Note: the value of KAFKA_ADVERTISED_HOST_NAME above should be changed to match the IP address of the kafka-cluster machine. e.g. the output of:
$ docker-machine ip kafka-cluster
Also, we are creating a topic named test with 10 partitions and 1 replica using the KAFKA_CREATE_TOPICS parameter. We will use this topic below for testing.
Now, start the containers using the docker-compose command.
# Note: this may take a few minutes to complete
$ docker-compose -f docker-compose-kafka-cluster.yml up -d
If everything goes well, you should be able to telnet to the broker and Zookeeper node.
$ telnet $(docker-machine ip kafka-cluster) 9092
$ telnet $(docker-machine ip kafka-cluster) 2181
Now that we are running a local Kafka cluster using Docker, we can test everything out using kafkacat. First, lets produce some messages to our test topic. After running the kafkacat command below, type a couple of words and hit enter (e.g. hello and world)
$ kafkacat -P -b $(docker-machine ip kafka-cluster):9092 -t test
hello
world
^C
When you are finished, interrupt the process (ctrl + c). Finally, flip the -P flag to a -C in the command above to run kafkacat in consumer mode, and make sure we can consume the records that were just written to the test topic. You should see hello and world (or whatever lines you typed) appear in the output.
kafkacat -C -b $(docker-machine ip kafka-cluster):9092 -t test
# output
hello
world
This completes our tutorial on how to set up a local Kafka cluster using Docker. This tutorial will come in handy as we explore additional topics related to Kafka in the future. Please stay tuned for new tutorials, and thank you for reading.