Kafka on Mac (2)

Now, let's do some python with our kafka. (Hey, your kafka and zookeeper should still be running.) (and if not, start 'em up again)

in a directory named (lines10) in your projects somewhere, create a file producer.py

from time import sleep
from json import dumps
from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],
    value_serializer=lambda m: dumps(m).encode('ascii'))

for e in range(1000):
    data = {'number' : e}
    producer.send('test', value=data)

And in consumer.py

from kafka import KafkaConsumer
from json import loads

consumer = KafkaConsumer(
     value_deserializer=lambda m: loads(m.decode('ascii')))

for message in consumer:
    message = message.value
    print('{} found'.format(message))

Now, in two different terminal window, run producer.py in one, and consumer.py in the other. Watch them!

And you can start either (2 or more producers) or (2 or more consumers) across Kafka, and so you get lots of interesting combinations.