Rate-limit Kafka event generation with kcat and bash

Traffic for event streams

Recently, I worked with IBM Cloud Event Streams which is a message bus built with Apache Kafka. I was looking for a simple command-line tool to test my Event Streams instance and to stream access logs into it. That's when I ran into kcat (formerly known as kafkacat). It is a generic command line Kafka producer and consumer and easy to install - just use a Docker image. All worked well, I could even read a file of historic Apache access logs and, line by line, send them over. But I still faced the issue of controlling how much to send, how to throttle it. I solved it using a bash script.

Get educated: The IDUG Virtual 2021 EMEA Db2 Tech Conference is coming up

The virtual Db2 conference is coming up

Do you remember meeting in person, at a conference? Well, hang on to that memory because the International Db2 User Group (IDUG) conference for EMEA is going to be virtual again this year. The "IDUG Virtual 2021 EMEA Db2 Tech Conference" is scheduled for December 6-10. If you have not registered, I recommend doing it. Here are some highlights and links to explore further.

On serverless data scraping, cloud object storage, MinIO and rclone

Building a data lake the serverless way

This fall, I started another side project. It involves mobility data and its analysis and visualization. As first step and still ongoing, I am building up a data lake (and maybe integrating it into a data fabric). For that, I regularly download data from various sources, then upload it to a Cloud Object Storage/S3 bucket. Once in cloud storage, I process it further, often involving tools like the MinIO client and rclone. Let me give you a quick overview and links to some more background reading...

Db2 SQL PL book as PDF and many useful doc links

Tell the world about Db2 documentation

Yesterday was one of those days when I, again, worked as information broker. I received a question, asked around myself, passed on information that I learned, and finally handed back an answer to the originator. With this blog post, I am closing the loop and share what I learned. A, I forgot, it is about the Db2 documentation.

IBM Cloud resource reclamations: Some more details and my best practices

Reclaim resources

This is a quick follow-up to my "To restore or delete" post in which I discussed IBM Cloud resource reclamations and shared some tricks. Today, I have some more details and also want to share with you my best practices on handling resources.

Db2 on Cloud credentials and how to connect from your app

Over the past 12 months, Db2 on Cloud migrated instances to a new deployment model. It seems to be based on the IBM Cloud Databases model and resulted in a different structure for the service credentials. Previously, Db2 on Cloud instances were accessible on the typical ports 50000 and 50001. The JSON-based credentials object included ready-to-use connection URIs. Now, after the change, the VCAP object has more parts, but is missing the easy-to-use URIs. In the following, I am showing you how to consume the new credential structure in your app and to successfully connect to Db2 on Cloud.

To restore or to delete: Working with IBM Cloud resource reclamation

Resource management

Over the past weeks, I provisioned many resources in my IBM Cloud account. Typical to cloud computing, I also deleted and re-provisioned service instances. As part of that work, I dealt with resource reclamation and sometimes restored services or made sure they were "hard deleted", immediately gone for good.

In this blog post, I am going to give a brief overview of resource reclamation in IBM Cloud and share some tips and tricks from my experience.

Serverless Twitter Bot using IBM Cloud

A Twitterbot at work

Some months ago, I discussed cron-like scheduling on IBM Cloud. One of my personal use cases is to send out tweets. In this post, I am going to look into details, into how I implemented a Twitter bot, deployed it to IBM Cloud Code Engine, and how it is managed.

Password expiration and vacation planning

Ready for vacation: Passwords

The months of July and August are typical vacation times. Many people work with (at least mental) packing list to prepare for some time off. On my list are passwords. I can relax knowing that no password will expire when I am away. The reason is that for some systems it is a big hassle to reset expired password or accounts. Thus, I prepare accounts for vacation.

How to connect from Python to Db2

One of the recurring questions I have seen is "how to connect from Python to Db2". I have blogged about Python and Db2 a couple times before. Recently, the question has been popping up again more frequently. From my view, the increased security with mandatory SSL/TLS encryption and the use of Python-based Jupyter notebooks for data science and AI / ML projects are the drivers behind that increase. Moreover, there is Db2 on-premises and the Db2 on Cloud / Db2 Warehouse on Cloud plus container-based offerings. Today, I am trying to sort this out and answer that question. Again.... :)

Cloud Security: BYOK vs. KYOK explained

Keep and bring your own key

When talking about cloud security and key management systems (KMS) for data encryption, we often hear the terms BYOK and KYOK. But what do they mean and what is the difference? Let me try to explain in this quick write-up. BYOK stands for "bring your own key" and refers to the ability to import an existing - your own - encryption root key into a (cloud-based) key management system.

Cloud-based HSM with PKCS #11 for Db2 Native Encryption

Manage your encryption keys

When using Db2 databases, there are a different ways to encrypt the stored data (data at rest). One of them is to use the built-in feature, Db2 native encryption. It supports three kinds of keystores, a local keystore file or centralized key management systems (KMS) based on KMIP or PKCS #11 protocols.

Today, I want to point you to a tutorial in the IBM Cloud documentation. It discusses using Hyper Protect Crypto Services PKCS #11 for Db2 native encryption.

Db2 11.5.6 is available

You probably already have noticed that a new release of Db2 for Linux, UNIX, and Windows is available, Db2 11.5.6. You can download the Db2 Fix Pack via the usual support site. The related documentation highlights the following features:

• Improved high availability with Advanced Log Space Management,
• Graph modeling and analysis of Db2 data using IBM Db2 Graph,
• Restrictions lifted on accessing column-organized tables
• Technical preview update to Machine Learning Optimizer
• New Click-to-Containerize utility

Aside from the highlights page, I usually go over the enhancements by category. Here are my personal highlights:

• The SQL enhancements include new NOWAIT and WAIT clauses for SELECT, UPDATE and DELETE statements. You can now specify at the statement level how many seconds to wait for locks. 
• High availability, backup, resiliency, and recovery enhancements lists improvement to ADMIN_MOVE_TABLE and also has news on pacemaker. 
• And in terms of Workload management enhancements, you can now enable the Adapte Workload Manager.

I am sure that we are going to learn all the details at the IDUG EMEA 2021 conference in Edinburgh, Scotland, in October. Mark your calendars.

Quickly deploy the serverless cloud mailer using Terraform

In the era of instant messaging we all still receive emails. They are used for status updates, security alerts or just for proposing really great offers. Recently, I blogged about how to have the IBM Cloud Security Advisor send out alerts using your SMTP-based email delivery service. Later, I made the solution core, a serverless action available as separate project "cloudmailer" on GitHub and blogged about it: A Serverless Function for Sending Emails on IBM Cloud. Continuing this side project, I now added Terraform support. Thus, using "terraform apply" you can now automatically deploy everything including the SMTP configuration. See the instructions in code repository for details.

IBM Cloud: Send out alert emails using SMTP

Last year, I blogged about how to use Slack or email notifications for security IBM Cloud security issues. Now I added another code sample, showing how to send out notification emails using any SMTP server. The code is written in Python and Node.js and deployed as IBM Cloud Functions-based webhook. The core part in charge of connecting to an SMTP email server and sending the email is isolated as dedicated action. Thus, it is possible to use it on its own, e.g., to send out emails unrelated to the IBM Cloud Security & Compliance Center.

My best practices: How to search Db2 documentation

I often answer technical product questions, for Db2 and IBM Cloud topics. To provide relevant links and to back up my "hunch" after reading a question, I typically search the relevant documentation. But what are efficient way to search in the Db2 documentation? What are good ways to find the relevant parts in the IBM Cloud documentation? Here are my best practices on searching documentation.

Wireshark with Lua on RHEL / CentOS

Wireshark with Lua-based dissector

What do you do on a rainy public holiday with COVID19 restrictions in place? Finally get Wireshark to work with Lua support to have custom dissectors. Dissectors are useful to turn binary garbage into readable TCP or UDP packet content. Lua is a scripting language and a supported way of adding dissectors in Wireshark. Unfortunately, the install package for Red Hat Enterprise Linux does not include Lua support. Compiling Wireshark on my RHEL 8.3 does not simply work because it requires Lua version 5.2 for my scripts to work. And RHEL either has version 5.3 or 5.1 which both are incompatible (long story). So, let's get going.

Upcoming Db2 IDUG conferences

A quick post on the upcoming Db2 conferences, organized by the International Db2 User Group (IDUG). 

• The IDUG North America "2021 NA Db2 Tech Conference" is going to be a virtual event again. It is scheduled from June 7-16, 2021. The conference agenda is already available and registration is open.
• The European IDUG conference, 2021 EMEA Db2 Tech Conference, is still slated to be an in-person event. I hope to meet you in Edinburgh, Scotland, this year in October. The conference team offers mentoring for new speakers. The submission deadline just passed, but if you are a customer with a good proposal, there might be .... 

Last, but not least, I recommend to take a look at the IDUG page with regional Db2 user groups. Some of them offer virtual events, too, and facilitating networking and exchange of information during the pandemic.

cron-like scheduling on IBM Cloud

cron-like scheduling

Some days ago I stumbled over my 2015 post "Bluemix: Simple cron-like service for my Python code". It is not just the name Bluemix which is dated. Since then, it has transformed into IBM Cloud Platform and has added serverless compute options like IBM Cloud Functions (OpenWhisk) and recently IBM Cloud Code Engine. Both of them support "eventing", event-based execution of code. And both support Python code as well as many other programming languages like Node.js, Go (Golang), PHP and more. So, what does it take to set up cron-like scheduling? Not much.

Cloud tutorial on serveless web app and eventing

A follow-up from my last post on Python decorators:

Solution architecture

Some years back, on IBM Cloud, we introduced an IBM Cloud solution tutorial for GitHub traffic analytics based on Cloud Foundry and IBM Cloud Functions. A Cloud Functions action is triggered daily to collect traffic data. The action stores the data in a Db2 database. Users can then analyse the data in a Python Flask app served by Cloud Foundry.

Today, that same solution scenario and app are still available, but they are served by IBM Cloud Code Engine. Code Engine is a fully managed, serverless platform that runs your containerized workloads, including web apps, microservices, event-driven functions or batch jobs. The slightly renamed tutorial — "Serverless web app and eventing for data retrieval and analytics" — demonstrates how the existing app can be containerized and both served as web app and and used to process the daily data collection event.

Continue reading the article on the IBM Cloud blog.

Pseudo-decorators for my Python Flask app

Secured Python Flask app on Code Engine

Recently, I migrated an existing Python app from a Cloud Foundry to an IBM Cloud Code Engine deployment. The Flask app uses decorators for the routes and for OIDC-based authentication. For usability, the app should start up even without anything configured yet and the OIDC decorators being invalid. It required to some IMHO tricky coding.

JWT token authentication in Db2 runtimes like Python or Node.js

Python script connecting to Db2 with JWT

Some weeks ago I discussed how to configure JWT-based token authentication in Db2. I set up Db2 to accept JWS identity tokens and then connected to my test database using the command line. But how do you connect from a programming language like Python or Node.js? Here is what I needed.

Great chatbots in no time

Chatbots take over customer service

Last week, I was coach again at a chatbot hackathon. Chatbots have been around for a long time and I have blogged about tips & tricks for developing chatbots and resources many times. With Covid-19 transforming the world towards more digitalization and self-service, chatbot deployments grew significantly. In this post, I want to look back at the hackathon and share some wisdom for building great chatbots.

IBM Cloud CLI: Some tips and tricks - make it a home

IBM Cloud CLI options

In the past, I have blogged several times about how to use the command line. Today, I want to share some tips and tricks for the IBM Cloud CLI (command line interface). Many of you work with the ibmcloud command, but have you ever looked into its help option? Here is a quick look at them and how to put them to good use...

Follow-up to Db2 and JWT: What is JOSE...?

JWT: Encoded or decoded security claims

Last week, I wrote about Db2 support for JSON Web Tokens (JWT). Today, I have a small follow-up with some reading material on JWT and related topics like JOSE, JWS, JWK as well as OAuth and OpenID Connect.

Some JWT history and standards

The first draft for JSON Web Tokens, JWTs, is already 10 years old. It is from December 2010. The early draft states:

Db2 Security: Configure JSON Web Token (JWT) authentication

Db2 login utilizing a JWT

Since V11.5.4, Db2 allows to consume JWT (JSON Web Token). The business case is to support SSO (single sign-on). You can configure Db2 to accept access tokens issued by external Identity Provider (IDPs), including the IBM solutions or open source projects like gluu. V11.5.5 brought some JWT improvements, e.g., support for multiple labels. On a recent Friday, I set up my Db2 Docker container to accept JWTs, not those issued by an external identity service, but my own.