This is a blog post from our Community Stream: by developers, for developers. Don’t miss to stop by our community to find similar articles or join the conversation.
Before we find out why the Kafka rocket can fly at all and how it does so, we ignite the engines together and take a short test flight:
For our flight into space, we launch our (of course reusable) rocket, briefly leave the atmosphere to practice re-entry, and gently and safely land the rocket on a landing pad.
We want to capture all flight phases in Kafka and create a topic first. Topics are similar to tables in databases in that we store a collection of data of a certain kind in a topic. In our case, it is flight data, so we call the topic flightdata:
$ kafka-topics.sh \
We use the kafka-topics.sh flightdata --topic flightdata --create --partitions 1 --replication-factor 1 kafka-topics.sh 9092 --bootstrap-server localhost:9092
So now we have a place to store our data. The rocket’s board computer continuously sends us updates on the rocket’s flight state. For our simulation, we use the command line tool kafka-console-producer.sh --topic parameter countdown started into our just created topic flightdata
$ echo "Countdown started" | kafka-console-producer.sh \
Our ground station now wants to read this data and output it on a large screen so that we can see whether the rocket really works as we expect it to. Let’s take a look at what has happened so far. To read our sent message again, we start the kafka-console-consumer.sh
$ timeout 10 kafka-console-consumer.sh \
When we start the kafka-console-consumer.sh --topic flightdata
Somewhat surprisingly, no message is displayed. This is because by default, the kafka-console-consumer.sh
$ timeout 10 kafka-console-consumer.sh \
This time we see the message Countdown started ! So, what happened? We used the kafka-topics.sh flightdata kafka-console-producer.sh kafka-console-consumer.sh
Without any other information, the kafka-console-consumer.sh --from-beginning
Interestingly, respectively unlike in many messaging systems, we can read messages not only once, but as many times as we want. We can use this, for example, to connect several independent ground stations to the topic so that they can all read the same data. Or there may be different systems that all need the same data. We’re capable of imagining having not only a display screen, but in addition other services too, such as a service that compares the flight data with current weather data and decides if something needs to be done. However, we may also want to analyze the data after the flight and require the historical flight data for that. To accomplish this, we can simply run the Consumer multiple times and each time get the same result.
However, we would now like to display the current state of the rocket in our control center, in such a way that the display updates immediately when there is new data. For this, we start the kafka-console-consumer.sh
# Don't forget: use CTRL+C to stop the consumer
To simulate the producer on the rocket side, we now start the kafka-console-producer.sh CTRL+D EOF
# Don't forget: use CTRL+D to stop the Producer
The kafka-console-producer.sh sends one message to Kafka per line we write. That means we can now type messages into the terminal with the producer:
# Producer window
We should also see these promptly in the window with the consumer:
# Consumer Window:
Let’s imagine that part of our ground crew is in the home office and wants to follow the flight from home. To do so, they independently start their consumer. We can simulate this by starting a kafka-console-consumer.sh
# Window Consumer 2
We can see here that data, once written, can be read in parallel by multiple consumers without the need for the consumers to talk to each other or for the consumers to register with Kafka first. Kafka does not delete any data, which means that we can still start a consumer later that can read historical data.
Now, let’s say that we don’t want just one rocket to fly at a time, but several. Kafka has no problem with this and can easily process data from numerous Producers simultaneously. So let’s start another Producer in another terminal window:
# Producer for Rocket 2
We see all the news from all the Producers show up in all our Consumers in the order the news was produced:
# Consumer 1 window
But now the issue arises that we cannot distinguish the messages from rocket 1 and 2. We could possibly write in each message from which rocket the message was sent. But more about that later.
Before we continue, we should abort the flight of our second rocket, as we want to launch it later:
# Producer for Rocket 2
We have now successfully launched and landed a rocket on a test basis. In the process, we wrote some data in Kafka. To accomplish this, we first created a topic flightdata kafka-topics.sh kafka-console-producer.sh kafka-console-consumer.sh --from-beginning kafka-console-consumer.sh
After gaining this experience, we can now close all open terminals. We close Producer with CTRL+D CTRL+C German book and in upcoming Xeotek Community articles.
