Kafka on Mac (1)

Kristofer Younger -

So make sure you have java 1.8 running

$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

and then install kafka

brew install kafka

this should install two packages: zookeeper & Kafka

Now do either

pip install kafka-python

or

conda install -c conda-forge kafka-python

now it's time to start both zookeeper (which ends up on port 2181) and kafka (whihc will end up running on port 9092)

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

And now in a separate shell (I use two different tabs on the same terminal window)

kafka-server-start /usr/local/etc/kafka/server.properties

and then open yet a third new Tab in Terminal, for your python producer.
In this third tab, run

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

If that doesn't work, you might have to try:

kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

which creates a topic called "test"

Now, create two new terminal windows, once for producer, and one for consumer.
In producer, type

kafka-console-producer --broker-list localhost:9092 --topic test

Goodness happens if you get a ">" on a line by itself.
Let's look at that line - the command, kafka-console-producer - a kafka producer that runs on the command line. localhost:9092 - the machine and port where messages should get sent. and finally, --topic test - which topic to send the messages to inside of kafka.

Type in a message: Hello Wolrd! and return.
You should get another ">", now type No, I meant World! and return... and type a series of, say, 5 more lines of messages. Mne ended up looking like this

$ kafka-console-producer --broker-list localhost:9092 --topic test
>Hello Wolrd!
>No, I meant World!
>wow this is kinda cool
>and now what do I do?
>I'm bored
>and one more
>last message
>

These messages are getting stacked up inside of Kafka. Now, we need to run another program to pull them out of Kafka.
In the other terminal window, consumer, type

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

And see the --from-beginning? Meaning, grab messages in the log and start with the first message.
Mine looked like this:

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
Hello Wolrd!
No, I meant World!
wow this is kinda cool
and now what do I do?
I'm bored
and one more
last message

If you have the two windows side by side, as you type into the producer, you'll see it come out on the consumer.

This is the second time I ran the producer and consumer. The consumer started over from the beginning, and re-read all the messages I had sent the first session. Because why? Kafka's log is persisent unless you explicitly remove the messages...

Now, why is that?

and finally, lookee here: Kafka Information