Setting up a Kafka on AWS EC2

Setting up a Kafka on AWS EC2

March 26, 2021 / Nirav S

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java.

Updating System Packages

sudo apt-get update && sudo apt-get upgrade

Create a User for Kafka

sudo useradd kafka -m

Add a password to the Kafka user using passwd:

sudo passwd kafka

Next, we need to add the kafka user to the sudo group for it to have sudo privileges that will be required to install Kafka’s dependencies:

sudo adduser kafka sudo

You can now log in to the account, as shown below:

su -l kafka

Download and Extract the Kafka Binaries

mkdir ~/Downloads
curl "https://www.apache.org/dist/kafka/2.1.1/kafka_2.11-2.1.1.tgz" -o ~/Downloads/kafka.tgz

Next, we create a directory called kafka

mkdir ~/kafka

Change our working directory to this directory:

cd ~/kafka

Extract the archive we downloaded using the tar command:

tar -xvzf ~/Downloads/kafka.tgz --strip 1

Configure the Kafka Server

Kafka’s configuration options are specified in the server.properties file. Open this file with nano or your favourite editor:

nano ~/kafka/config/server.properties

Add the following line to the bottom of the file to allow us to delete Kafka topics.

Setting up a Kafka on AWS EC2

delete.topic.enable = true

Save and close the file.

Next, start Kafka with the command below:

sudo systemctl start kafka
journalctl -u kafka

The Kafka server listens on port 9092.

To enable Kafka on server boot, issue the command below:

sudo systemctl enable kafka

So we have successfully installed Kafka on the AWS.

Talk to AWS Certified Consultant

    Let Us Talk About Your AWS Development Requirements

    Have queries about your AWS project ideas and concepts? Please drop in your project details to discuss with our AWS experts, professionals and consultants.

    • Swift Hiring and Onboarding
    • Experienced and Trained AWS Team
    • Quality Consulting and Programming
    Let’s Connect and Discuss Your Project


      ">