So make sure you have java 1.8 running
$ java -version java version "1.8.0_66" Java(TM) SE Runtime Environment (build 1.8.0_66-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
and then install kafka
brew install kafka
this should install two packages: zookeeper & Kafka
Now do either
pip install kafka-python
conda install -c conda-forge kafka-python
now it's time to start both zookeeper (which ends up on port 2181) and kafka (whihc will end up running on port 9092)
And now in a separate shell (I use two different tabs on the same terminal window)
and then open yet a third new Tab in Terminal, for your python producer.
In this third tab, run
kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
If that doesn't work, you might have to try:
kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
which creates a topic called "test"
Now, create two new terminal windows, once for producer, and one for consumer.
In producer, type
kafka-console-producer --broker-list localhost:9092 --topic test
Goodness happens if you get a ">" on a line by itself.
Let's look at that line - the command, kafka-console-producer - a kafka producer that runs on the command line. localhost:9092 - the machine and port where messages should get sent. and finally, --topic test - which topic to send the messages to inside of kafka.
Type in a message:
Hello Wolrd! and return.
You should get another ">", now type
No, I meant World! and return... and type a series of, say, 5 more lines of messages. Mne ended up looking like this
$ kafka-console-producer --broker-list localhost:9092 --topic test >Hello Wolrd! >No, I meant World! >wow this is kinda cool >and now what do I do? >I'm bored >and one more >last message >
These messages are getting stacked up inside of Kafka. Now, we need to run another program to pull them out of Kafka.
In the other terminal window, consumer, type
kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
And see the
--from-beginning? Meaning, grab messages in the log and start with the first message.
Mine looked like this:
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning Hello Wolrd! No, I meant World! wow this is kinda cool and now what do I do? I'm bored and one more last message
If you have the two windows side by side, as you type into the producer, you'll see it come out on the consumer.
This is the second time I ran the producer and consumer. The consumer started over from the beginning, and re-read all the messages I had sent the first session. Because why? Kafka's log is persisent unless you explicitly remove the messages...
Now, why is that?
and finally, lookee here: Kafka Information