Today, we will be exploring how to set up a local Kafka cluster using Docker. This article will be referenced by future tutorials that require a working Kafka cluster. There are a few good tutorials already about how to set up a Kafka cluster locally, but I wasn’t completely satisfied with any of them. This is my attempt at covering the process in hopes that it will help someone else who is interested in both Docker and Kafka to create a usable development environment in just a few minutes.
The following items will need to be installed for this tutorial.
# Verify that Docker is installed $ docker --version Docker version 1.12.5, build 7392c3b
# Verify that VirtualBox is installed $ vboxmanage -v 5.0.10r104061
# Verify that kafkacat is installed $ which kafkacat /usr/local/bin/kafkacat
Note: if you would like to use another provider other than VirtualBox (e.g. VMWare Fusion, Digital Ocean, Amazon Web Services, etc), you can forgo the Virtual box installation above, and follow the directions for using Docker Machine with your chosen provider.
The first thing we need to do is create a Docker host using the
$ docker-machine create --driver virtualbox kafka-cluster
Once the above command has finished executing, you can verify that the machine was created by running:
$ docker-machine ls
You should see output similar to the following:
# output NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS kafka-cluster - virtualbox Running tcp://192.168.99.100:2376 v1.12.5
The IP address in the
URL column above will be needed for later steps. This will be the IP of our broker and Zookeeper node. Another way to retrieve this value is to run the
docker-machine ip command. For example:
$ docker-machine ip kafka-cluster # 192.168.99.100
Now, in order to set the kafka-cluster host as the active environment, we need to use the
docker-machine env command. This basically tells Docker which host to run the containers on when we bring them online in the steps below.
$ eval $(docker-machine env kafka-cluster)
Now that we have created a dedicated Docker host for our Kafka cluster, lets run a broker and Zookeeper node locally using the popular wurstmeister/kafka-docker image. First, clone the project using the following command:
$ git clone firstname.lastname@example.org:wurstmeister/kafka-docker.git $ cd kafka-docker
Now, within the
kafka-docker directory, add the following configuration to a file named
version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka ports: - "9092:9092" environment: KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100 KAFKA_CREATE_TOPICS: "test:10:1" KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock
Note: the value of
KAFKA_ADVERTISED_HOST_NAME above should be changed to match the IP address of the kafka-cluster machine. e.g. the output of:
$ docker-machine ip kafka-cluster
Also, we are creating a topic named
test with 10 partitions and 1 replica using the
KAFKA_CREATE_TOPICS parameter. We will use this topic below for testing.
Now, start the containers using the
# Note: this may take a few minutes to complete $ docker-compose -f docker-compose-kafka-cluster.yml up -d
If everything goes well, you should be able to telnet to the broker and Zookeeper node.
$ telnet $(docker-machine ip kafka-cluster) 9092 $ telnet $(docker-machine ip kafka-cluster) 2181
Now that we are running a local Kafka cluster using Docker, we can test everything out using kafkacat. First, lets produce some messages to our
test topic. After running the
kafkacat command below, type a couple of words and hit enter (e.g.
$ kafkacat -P -b $(docker-machine ip kafka-cluster):9092 -t test hello world ^C
When you are finished, interrupt the process (ctrl + c). Finally, flip the
-P flag to a
-C in the command above to run kafkacat in consumer mode, and make sure we can consume the records that were just written to the
test topic. You should see
world (or whatever lines you typed) appear in the output.
kafkacat -C -b $(docker-machine ip kafka-cluster):9092 -t test
# output hello world
This completes our tutorial on how to set up a local Kafka cluster using Docker. This tutorial will come in handy as we explore additional topics related to Kafka in the future. Please stay tuned for new tutorials, and thank you for reading.