Posts /

Deploying a local Kafka cluster with Docker

08 Jan 2017


Today, we will be exploring how to set up a local Kafka cluster using Docker. This article will be referenced by future tutorials that require a working Kafka cluster. There are a few good tutorials already about how to set up a Kafka cluster locally, but I wasn’t completely satisfied with any of them. This is my attempt at covering the process in hopes that it will help someone else who is interested in both Docker and Kafka to create a usable development environment in just a few minutes.


The following items will need to be installed for this tutorial.

# Verify that Docker is installed
$ docker --version
Docker version 1.12.5, build 7392c3b
# Verify that VirtualBox is installed
$ vboxmanage -v
# Verify that kafkacat is installed
$ which kafkacat

Note: if you would like to use another provider other than VirtualBox (e.g. VMWare Fusion, Digital Ocean, Amazon Web Services, etc), you can forgo the Virtual box installation above, and follow the directions for using Docker Machine with your chosen provider.

Create a Docker host

The first thing we need to do is create a Docker host using the docker-machine command.

$ docker-machine create --driver virtualbox kafka-cluster

Once the above command has finished executing, you can verify that the machine was created by running:

$ docker-machine ls

You should see output similar to the following:

# output
NAME            ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
kafka-cluster   -        virtualbox   Running   tcp://           v1.12.5

The IP address in the URL column above will be needed for later steps. This will be the IP of our broker and Zookeeper node. Another way to retrieve this value is to run the docker-machine ip command. For example:

$ docker-machine ip kafka-cluster 

Now, in order to set the kafka-cluster host as the active environment, we need to use the docker-machine env command. This basically tells Docker which host to run the containers on when we bring them online in the steps below.

$ eval $(docker-machine env kafka-cluster)

Run Kafka and Zookeeper locally

Now that we have created a dedicated Docker host for our Kafka cluster, lets run a broker and Zookeeper node locally using the popular wurstmeister/kafka-docker image. First, clone the project using the following command:

$ git clone 
$ cd kafka-docker

Now, within the kafka-docker directory, add the following configuration to a file named docker-compose-kafka-cluster.yml.

version: '2'
    image: wurstmeister/zookeeper
      - "2181:2181"
    image: wurstmeister/kafka
      - "9092:9092"
      KAFKA_CREATE_TOPICS: "test:10:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      - /var/run/docker.sock:/var/run/docker.sock

Note: the value of KAFKA_ADVERTISED_HOST_NAME above should be changed to match the IP address of the kafka-cluster machine. e.g. the output of:

$ docker-machine ip kafka-cluster

Also, we are creating a topic named test with 10 partitions and 1 replica using the KAFKA_CREATE_TOPICS parameter. We will use this topic below for testing.

Now, start the containers using the docker-compose command.

# Note: this may take a few minutes to complete
$ docker-compose -f docker-compose-kafka-cluster.yml up -d

If everything goes well, you should be able to telnet to the broker and Zookeeper node.

$ telnet $(docker-machine ip kafka-cluster) 9092
$ telnet $(docker-machine ip kafka-cluster) 2181


Now that we are running a local Kafka cluster using Docker, we can test everything out using kafkacat. First, lets produce some messages to our test topic. After running the kafkacat command below, type a couple of words and hit enter (e.g. hello and world)

$ kafkacat -P -b $(docker-machine ip kafka-cluster):9092 -t test

When you are finished, interrupt the process (ctrl + c). Finally, flip the -P flag to a -C in the command above to run kafkacat in consumer mode, and make sure we can consume the records that were just written to the test topic. You should see hello and world (or whatever lines you typed) appear in the output.

kafkacat -C -b $(docker-machine ip kafka-cluster):9092 -t test
# output

What now?

This completes our tutorial on how to set up a local Kafka cluster using Docker. This tutorial will come in handy as we explore additional topics related to Kafka in the future. Please stay tuned for new tutorials, and thank you for reading.