Publish Messages to Kafka Topic from Python

Read Messages from kafka topic using Python

In this post, I will show you how to read messages from a Kafka Topic using Python.

Prerequisites

• Kafka Cluster
• Kafka Topic
• Python

Install Confluent Kafka package

Install the confluent-kafka package using the following command:

pip install confluent-kafka

Read messages from Kafka Topic

Add the following code in read_messages_from_kafka_topic.py file to read messages from the Kafka Topic:

from confluent_kafka import Consumer, KafkaException

def read_messages_from_topic(bootstrap_servers, topic_name):
    consumer = Consumer({
        'bootstrap.servers': bootstrap_servers,
        'group.id': 'my-group',
        'auto.offset.reset': 'earliest'
    })

    consumer.subscribe([topic_name])

    try:
        while True:
            message = consumer.poll(timeout=1.0)

            if message is None:
                continue
            if message.error():
                if message.error().code() == KafkaError._PARTITION_EOF:
                    continue
                else:
                    raise KafkaException(message.error())
            else:
                print(f'Message: {message.value().decode("utf-8")}')
    except KeyboardInterrupt:
        consumer.close()

if __name__ == '__main__':
    bootstrap_servers = 'localhost:9092'
    topic_name = 'test-topic'

    read_messages_from_topic(bootstrap_servers, topic_name)

Run the Python script

Run the Python script using the following command:

python read_messages_from_kafka_topic.py

That’s it! You have successfully read messages from a Kafka Topic using Python.

Conclusion

In this post, I have shown you how to read messages from a Kafka Topic using Python. You can use the confluent-kafka package to interact with Kafka from Python.

I hope you find this post helpful. Please let me know if you have any questions or feedback in the comments section below. Thank you for reading!