Read Messages from kafka topic using Python#
In this post, I will show you how to read messages from a Kafka Topic using Python.
Prerequisites#
- Kafka Cluster
- Kafka Topic
- Python
Install Confluent Kafka package#
Install the confluent-kafka package using the following command:
1
pip install confluent-kafka
Read messages from Kafka Topic#
Add the following code in read_messages_from_kafka_topic.py file to read messages from the Kafka Topic:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from confluent_kafka import Consumer, KafkaException
def read_messages_from_topic(bootstrap_servers, topic_name):
consumer = Consumer({
'bootstrap.servers': bootstrap_servers,
'group.id': 'my-group',
'auto.offset.reset': 'earliest'
})
consumer.subscribe([topic_name])
try:
while True:
message = consumer.poll(timeout=1.0)
if message is None:
continue
if message.error():
if message.error().code() == KafkaError._PARTITION_EOF:
continue
else:
raise KafkaException(message.error())
else:
print(f'Message: {message.value().decode("utf-8")}')
except KeyboardInterrupt:
consumer.close()
if __name__ == '__main__':
bootstrap_servers = 'localhost:9092'
topic_name = 'test-topic'
read_messages_from_topic(bootstrap_servers, topic_name)
Run the Python script#
Run the Python script using the following command:
1
python read_messages_from_kafka_topic.py
That’s it! You have successfully read messages from a Kafka Topic using Python.
Conclusion#
In this post, I have shown you how to read messages from a Kafka Topic using Python. You can use the confluent-kafka package to interact with Kafka from Python.
I hope you find this post helpful. Please let me know if you have any questions or feedback in the comments section below. Thank you for reading!