Installing and Running Apache Kafka on Ubuntu: A Step-by-Step Guide: Part 2

What is Apache Kafka?

Apache Kafka is a powerful distributed streaming platform that many organizations use for building real-time data pipelines and streaming applications. If you’re new to Kafka and want to set it up on Ubuntu, this guide will take you through the installation process and teach you how to start Zookeeper, run the Kafka server, create topics, produce messages, and consume messages.

Apache Kafka In Ubuntu

Key Features of Apache Kafka:

• Scalability: Kafka can handle large amounts of data and scale out easily.
• Fault Tolerance: Kafka replicates data across multiple brokers to ensure high availability.
• High Throughput: Kafka can process millions of messages per second, suitable for real-time use cases.
• Durability: Kafka persists messages to disk, ensuring data is stored reliably.

Kafka Architecture:

At a high level, Kafka is composed of the following components:

• Producer: Sends data (messages) to Kafka topics.
• Consumer: Reads data from Kafka topics.
• Broker: A Kafka server that stores and serves data.
• Topic: A logical channel to which producers send messages and from which consumers read.
• Zookeeper: Manages and coordinates Kafka brokers, ensuring proper operation.

With this understanding of Kafka, let’s get started with the installation on Ubuntu.

Step 1: Installing Java (JDK)

Kafka requires Java to run. If you don’t already have Java installed, you can install the OpenJDK package:

Update your package list:

sudo apt update

Install OpenJDK:

sudo apt install openjdk-11-jdk -y

Verify the installation:

java -version

You should see an output displaying the Java version installed.

Step 2: Download and Extract Apache Kafka

Download the latest version of Kafka from the official website. You can use wget to download Kafka directly:

wget https://archive.apache.org/dist/kafka/3.4.0/kafka_2.12-3.4.0.tgz

Extract the downloaded tar file:

tar -xzf kafka_2.12-3.4.0.tgz

Move the extracted folder to /usr/local/kafka:

sudo mv kafka_2.12-3.4.0 /usr/local/kafka

Step 3: Start Zookeeper

Kafka uses Zookeeper for managing and coordinating Kafka brokers. To start Zookeeper:

cd /usr/local/kafka

Start Zookeeper using the default configuration:

sudo bin/zookeeper-server-start.sh config/zookeeper.properties

This will start Zookeeper, and you should see logs indicating that Zookeeper has started successfully.

Step 4: Start the Kafka Server

With Zookeeper running, you can now start the Kafka broker.

1. Open another terminal and navigate to the Kafka directory:

cd /usr/local/kafka

2. Start the Kafka broker:

sudo bin/kafka-server-start.sh config/server.properties

This will start the Kafka server using the default configuration. You should see logs indicating that the Kafka server is up and running.

Step 5: Create a Kafka Topic

In Kafka, data is organized into topics. A topic is a logical channel where producers send data and consumers read from it.

Open another terminal and navigate to the Kafka directory:

cd /usr/local/kafka

Create a new topic:

sudo bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

• --topic my-topic: Creates a topic named "my-topic".
• --bootstrap-server localhost:9092: Specifies the Kafka broker's address.
• --partitions 1: Creates one partition for the topic.
• --replication-factor 1: Sets the replication factor to 1 (no replication).

You should see a confirmation message indicating that the topic has been created successfully.

Step 6: Produce Messages to the Kafka Topic

Now that the topic is created, let’s send some messages to it using a Kafka producer.

Open another terminal and navigate to the Kafka directory:

cd /usr/local/kafka

Start the Kafka producer:

sudo bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092

This will start a console where you can type messages. Each line you enter will be sent as a message to the “my-topic” topic.

>Hello, Kafka!
>This is my first message.

Step 7: Consume Messages from the Kafka Topic

Finally, let’s read the messages you just produced using a Kafka consumer.

Open another terminal and navigate to the Kafka directory:

cd /usr/local/kafka

Start the Kafka consumer:

sudo bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092

--from-beginning: This tells the consumer to read messages from the beginning of the topic.

You should see the messages you produced earlier displayed in the console.

Conclusion

Congratulations! You’ve successfully installed and run Apache Kafka on Ubuntu. You also learned how to start Zookeeper, run the Kafka server, create topics, produce messages, and consume messages. While this guide covers the basics, Kafka has many advanced features, such as stream processing and distributed processing, that you can explore as you grow more familiar with the platform.

Whether you’re building a small-scale project or a large-scale system, Kafka’s scalability, fault tolerance, and high throughput make it an excellent choice for real-time data pipelines and event-driven applications.

If you encounter any issues or have questions, feel free to leave a comment below. Happy streaming with Kafka!