Rate-limit Kafka event generation with kcat and bash

Traffic for event streams

Recently, I worked with IBM Cloud Event Streams which is a message bus built with Apache Kafka. I was looking for a simple command-line tool to test my Event Streams instance and to stream access logs into it. That's when I ran into kcat (formerly known as kafkacat). It is a generic command line Kafka producer and consumer and easy to install - just use a Docker image. All worked well, I could even read a file of historic Apache access logs and, line by line, send them over. But I still faced the issue of controlling how much to send, how to throttle it. I solved it using a bash script.

Get educated: The IDUG Virtual 2021 EMEA Db2 Tech Conference is coming up

The virtual Db2 conference is coming up

Do you remember meeting in person, at a conference? Well, hang on to that memory because the International Db2 User Group (IDUG) conference for EMEA is going to be virtual again this year. The "IDUG Virtual 2021 EMEA Db2 Tech Conference" is scheduled for December 6-10. If you have not registered, I recommend doing it. Here are some highlights and links to explore further.

On serverless data scraping, cloud object storage, MinIO and rclone

Building a data lake the serverless way

This fall, I started another side project. It involves mobility data and its analysis and visualization. As first step and still ongoing, I am building up a data lake (and maybe integrating it into a data fabric). For that, I regularly download data from various sources, then upload it to a Cloud Object Storage/S3 bucket. Once in cloud storage, I process it further, often involving tools like the MinIO client and rclone. Let me give you a quick overview and links to some more background reading...

Db2 SQL PL book as PDF and many useful doc links

Tell the world about Db2 documentation

Yesterday was one of those days when I, again, worked as information broker. I received a question, asked around myself, passed on information that I learned, and finally handed back an answer to the originator. With this blog post, I am closing the loop and share what I learned. A, I forgot, it is about the Db2 documentation.