Analyze your IBM Cloud access management setup

ER diagram for cloud security data

Recently, I looked into how to analyze the access management setup of my IBM Cloud account. I wanted to better understand what kind of access policies exist, what service instances are covered by rules and policies, etc. In the past, I have shared with you how to get insights into IBM Cloud account privileges or how to improve security by identifying inactive identities. This time, I looked across the existing APIs to obtain identity and access management (IAM) and resource data. I retrieved, then analyzed that security data. With these insights, it is possible to improve security for your IBM Cloud account and its resources.

From Bluemix to IBM Cloud, from Cloud Foundry to Code Engine

"Bring Your Own Community"

About seven years ago, I started to work with, then blog about Bluemix and Cloud Foundry. Not my first, but one of the first posts is titled "Some fun with Bluemix, Cloud Foundry, Python, JSON and the Weather". Reading that article again I feel nostalgic and it brings back memories of how I learned to deploy my apps to Cloud Foundry. And how I had fun with new cloud technologies.

Cloud tutorial on serveless web app and eventing

A follow-up from my last post on Python decorators:

Solution architecture

Some years back, on IBM Cloud, we introduced an IBM Cloud solution tutorial for GitHub traffic analytics based on Cloud Foundry and IBM Cloud Functions. A Cloud Functions action is triggered daily to collect traffic data. The action stores the data in a Db2 database. Users can then analyse the data in a Python Flask app served by Cloud Foundry.

Today, that same solution scenario and app are still available, but they are served by IBM Cloud Code Engine. Code Engine is a fully managed, serverless platform that runs your containerized workloads, including web apps, microservices, event-driven functions or batch jobs. The slightly renamed tutorial — "Serverless web app and eventing for data retrieval and analytics" — demonstrates how the existing app can be containerized and both served as web app and and used to process the daily data collection event.

Continue reading the article on the IBM Cloud blog.

Pseudo-decorators for my Python Flask app

Secured Python Flask app on Code Engine

Recently, I migrated an existing Python app from a Cloud Foundry to an IBM Cloud Code Engine deployment. The Flask app uses decorators for the routes and for OIDC-based authentication. For usability, the app should start up even without anything configured yet and the OIDC decorators being invalid. It required to some IMHO tricky coding.

Db2 Security: Configure JSON Web Token (JWT) authentication

Db2 login utilizing a JWT

Since V11.5.4, Db2 allows to consume JWT (JSON Web Token). The business case is to support SSO (single sign-on). You can configure Db2 to accept access tokens issued by external Identity Provider (IDPs), including the IBM solutions or open source projects like gluu. V11.5.5 brought some JWT improvements, e.g., support for multiple labels. On a recent Friday, I set up my Db2 Docker container to accept JWTs, not those issued by an external identity service, but my own.

OBS on Linux: Green screen and virtual camera for video conferencing

OBS Studio: My monkey enjoys the beach

Similar to many of you, part of my work and hobbies consists of video conferencing. For some time now, I have been using OBS Studio (Open Broadcaster Software) to create a virtual camera on my Linux system. Recently, I had to upgrade my kernel. It required to recompile some file and reminded me that I wanted to blog about it. As usual, this is how I remember all the interesting stuff. So what is needed to create a virtual camera with OBS Studio and can you use a green screen for some beach feeling like shown?

Extend IBM Cloud Security Advisor with your own security metrics

Custom findings in Security Advisor

The IBM Cloud Security Advisor allows for centralized security management. It offers a unified dashboard that alerts security administrators for an IBM Cloud account of issues and helps them in resolving the issues. The advisor supports the integration of third-party vendors as well as custom findings. Using a REST API or programming language SDKs, it is possible to manage your own security metrics - from creating incident types and events to displaying them on the unified dashboard. In this and a follow-up post, I am going to give you a quick introduction to the Security Advisor and then discuss the code I made available on GitHub in the repo security-advisor-findings. It simplifies to management of custom objects and provides functions for some of my security scans.

Incorporate Git and IBM Cloud information into BASH command prompt

Many of you know that I love to use the command line. Because my day to day work includes interfacing IBM Cloud and GitHub, I have changed the BASH configuration to include related information into the command prompt. Here is what I did and how it looks like.

Automated reports with IBM Cloud Functions, Db2 and Slack

GitHub Traffic Analytics

One of my (many) favorite IBM Cloud solution tutorials is about combining serverless and Cloud Foundry for data retrieval and analytics. I blogged about it and described how an automated IBM Cloud Functions action retrieves GitHub statistics and stores them in Db2. Using an embedded Cognos dashboard and regular Javascript / HTML tables, the solution offers GitHub Traffic Analytics. I extended that solution by automatic weekly reports that are posted to Slack.

Now on GitHub: Understand and build chatbots the easy way

Recently, I posted about a then upcoming Meetup and my talk about chatbots. Here is a quick follow-up. To compile stuff for that presentation and some other upcoming talks, I created a GitHub repository "chatbot-talk2018". I has lots of links to get started and to deepen understanding around chatbot technology. Moreover, it contains a presentation in Markdown for GitPitch for you to use and extend. And finally, I wrote this brief introduction to some chatbot terms or concepts:

• Intents are what the user aims for, the desired action or result of the interaction. An intent can be to retrieve a weather report.
• Entities are (real or virtual) subjects or objects. For the example of the weather report, entities can be the city or country, e.g., Friedrichshafen in Germany, or date and time information such as "today afternoon".
• A dialog, dialog flow or dialog tree is used to structure the interaction. Typically, an interaction lasts longer than the user providing input and the chatbot returning a single answer. A dialog can be highly complex with several levels, subbranches, (directed) links between dialog nodes and more.
  For a weather chatbot, a dialog could be constructed that, after a greeting, asks the user about the location and time for a weather report, then asks if additional information, such as a weather outlook for the next few days, is needed.
• Slots are supported by several chatbot systems. Slots are used to specify the data items that need to be specified in order to produce the result of an intent. To return a weather report, e.g., at least the location and maybe the date or time is needed.
• Context is state information that is carried from step to step for a specific user interaction. The context typically stores the information that is already gathered as input (see "slot"), result-related data or metadata, or general chat information, e.g., the user name.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.

Extended: Manage and interact with Watson Assistant from the command line

Remember my blog posts about how to manage Watson Assistant from the command line and how to test context for a conversation? Well, that tool did not work well for server actions which I used in this tutorial on building database-driven Slackbot. The good news is that I found time to extend my command line Watson Conversation Tool to support credentials for IBM Cloud Functions.

With the recent update to the tool there are two new features:

1. Use the option "-outputonly" with the "-dialog" option to only print the output text, not the entire JSON response object. I introduced it to be able to demo dialog flows from the command line. Not everybody needs all the metadata for every dialog turn. Here is how it looks like when in action:



Chatbot dialog on the command line

2. In order to test dialog server actions, I need to provide the credentials for IBM Cloud Functions (ICF) in a private context variable. I recently blogged about how to enable the Watson botkit middleware for those server actions. For my tool, just provide the ICF key token as part of the configuration file. A sample is part of the GitHub repository.

For feature requests, defects or suggestions, open an issue or a pull request against the repository. If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.

Enable Botkit Middleware for Watson Assistant for serverless actions

Slack chatbot with Watson Assistant

Last week I blogged about using Botkit Middleware to create a Watson-powered database interface. I pointed to a tutorial demonstrating a Slack chatbot for a SQL database (Db2). I described that it is relatively simple to replace the Conversation connector with Botkit Middleware. Today, I want to point you to some code I wrote to enable the server-side (serverless) actions in Watson Assistant.

Automated, regular database jobs with IBM Cloud Functions (and Db2)

IBM Cloud Functions and Db2

Yesterday, I blogged about the latest tutorial I wrote. The tutorial discusses how to combine serverless and Cloud Foundry for data retrieval and analytics. That scenario came up when I looked into regularly downloading GitHub traffic statistics for improved usage insights. What I needed was a mechanism to execute small Python script on a daily or weekly basis. After looking into some possible solutions, IBM Cloud Functions was the clear winner. In this blog, I am going to discuss how simple it is to implement some regular, automated activities, such as maintenance jobs for a cloud database.

Use Db2 and IBM Cloud to analyze GitHub traffic data (tutorial)

Architecture: GitHub Traffic Analytics

In a new solution tutorial, I show you how to automatically retrieve and store GitHub traffic data the serverless way with IBM Cloud Functions and Db2. The data can then be analyzed via a Web app deployed to Cloud Foundry on IBM Cloud. The app is secured with App ID using OpenID Connect. The new service Dynamic Dashboard Embedded provides visualization of the views and clones of GitHub repositories.

EgoBot: Fun with a Slightly Mutating ChatBot

Fun with the Bluemix EgoBot

Over the past day and evening I had some fun with a slightly mutating chatbot. The API for the IBM Watson Conversation service offers REST calls to query and change the workspace, the parts that make up a chat. So why not try writing a chatbot that is egocentric? A chatbot that answers questions about itself, that is happy as long as everything is related to itself? Well, let me tell you about this fun project I call EgoBot.

The EgoBot is at an early stage right now. It supports queries about some of its metadata and adding new intents. And it has both an English and a German version (does language change its character...?). You can see a sample session below.

Chatting with the Bluemix EgoBot

To find out more about this chatbot head over to the EgoBot GitHub repository. The bot is written in Python and has everything to get you started with either an English or German conversation. Let me know about your Friday fun.

Chatbots: Manage Your Watson Conversations from the Command Line or App

Manage Watson Conversation Workspaces

I am a big fan of using the command line as most of you know by now. This applies to interacting with IBM Bluemix cloud and its services and for database systems like DB2 or dashDB. Thus, I was excited when the IBM Watson Conversation service added API functions to manage workspaces. To test the new API I wrote a small Python-based tool to manage my Conversation workspaces. It both demonstrates the API usage as well as gives you a nice command line tool to list your workspaces, update them, save local copies or even create or delete workspaces. Read on and learn how to manage your Conversation workspaces.

Write Your Own CLI Plugins for Bluemix Cloud Foundry

README for my Plugin

Last year I blogged about how I am using plugins to extend the Bluemix Cloud Foundry command line interface (CLI). The CLI has a set of commands to manage plugin repositories and to install and uninstall plugins. It is pretty easy to use and there are some useful plugins available from Cloud Foundry and IBM Bluemix. Having mastered the first step, I wanted to know how plugins work and what it takes to write my own plugin. Here is what I learned.

Context Path Routing of Apps and Services in Bluemix

Context Paths for Bluemix Apps

As I mentioned in my post yesterday about simplified deployment of complex apps, I have been working on a sample for Context Path Routing to be used with IBM Bluemix. But what are context path routes and what does the sample do? Here are the details.

Cloud Foundry introduced Context Path Routing last year. Until then there was the requirement that each app (or service) was served from its own hostname. Now, apps can share a host with each app being served from a specific path on that host. Here are two examples:

1. When building a larger website, there could be several so-called microsites embedded. With Context Path Routing it is possible to serve, e.g., example.com from one web app and example.com/user-management or example.com/news from other apps. All these apps could be written in different programming languages such as Node.js, Python, Java and others. 
2. For a more complex microservice-based app, following the principles of the Twelve Factor App, there could be several (backing) services involved. The app and each would require their own hostname. With Context Path Routing the app could use app.mybluemix.net and services could be served from app.mybluemix.net/service1, app.mybluemix.net/service2, etc.

So how could you use the new routing feature? There are several Cloud Foundry CLI commands related to routing. They allow to specify an additional path for application routes. Manifest files also have optional route properties that could be set. To get started, take a look at my Context Path Routing sample on GitHub for details. It has two small apps written in Python and Node.js that share a host using specific paths. The apps can be deployed with a single command and allow to experiment with context paths. The "What this samples does" section gives you some ideas.

Coincidence? CeBIT visitors and weather featuring Jupyter Notebooks, Spark and dashDB

Jupyter Notebook via Bluemix

Next week I am going to talk at the CeBIT fair in Hanover. As usual I am interested in how the weather will be. And with every conference or fair a common question is about attendance. Why not combine the two, analyse past CeBIT weather and visitor count for some Friday fun? Today I am going to look into Jupyter Notebooks on Apache Spark with some Open Data stored in dashDB, all available via IBM Bluemix.
(Note that I am in a hurry and don't have time for detailed steps today, but that I share the sources and will add steps later on.)

The screenshot on the right is the result of what I am going to produce today. The source file for the notebook, the exported HTML file, input data, etc. can be found in this GitHub repository. If you came here for DB2 or dashDB you might wonder what Jupyter Notebooks are. Notebooks are interactive web-pages where you have sections ("cells") that contain text or code. The text can be in different input formats including Markdown. The code cells support various programming languages, can be edited inline and are executed on demand. Basically a notebook is an interactive, on-demand business/database report. And as you can see in the screenshot, the code is able to produce graphs.

The IBM Analytics for Apache Spark service on Bluemix provides those analytic notebooks and it is the service I provisioned for my tests. Once you launch the service you can start off with sample notebooks or create them from scratch. I started with samples to get up to speed and the composed my own one (see my notebook source on GitHub). It has several cells written in Python to set up a connection to dashDB/DB2, execute queries, fetch data and process that data within the notebook. The data is used to plot out a couple graphs.

For my example I am using a dashDB (a DB2-based service) that I provisioned on Bluemix as a data store. I used the LOAD wizard to create and fill one table holding historic CeBIT dates and visitor counts and another table with historic weather data for Hanover, Germany (obtained from Deutscher Wetterdienst). Within the notebook those tables are queried and the data fetched into so-called data frames. The data frames are used to transform and shape the data as needed and as source for the generated graphs. Within the notebook it is possible to combine data frames, execute queries on them and more - something I didn't do today.

To get to my dashDB-based graphs in a Jupyter Notebook on IBM Analytics for Apache Spark I needed to get around some issues I ran into, including data type casts, naming of result columns, labeling of graphs, sourcing columns as input for a graph and more. For time reason I refer to the comments in the source code for my notebook.

After all that introduction, here is the resulting graph. It shows that during a sunny and warm week with close to no rain there were fewer CeBIT attendees. A little rain, some sun and average temperature yielded a high visitor count. So could it be that the weather to attendee relationship is bogus for computer fairs and may only hold for museums? Anyway, it was fun learing Jupyter Notebooks on Bluemix. Now I need to plot my weekend plans...

Historic CeBIT Weather and Attendance

Accessing DB2 from node.js on IBM Bluemix and locally

Some days ago I started experimenting with node.js. Other than JSON and some click functions on webpages I don't have much experience with JavaScript. The reason for this "adventure" is that node.js is offered as one of several programming languages on IBM Bluemix, IBM's platform-as-a-service (PaaS). I wanted to find out how complex or easy it would be to bring both node.js and DB2 (IBM's relational and in-memory database system) together.

When I start with some new language I typically produce errors. Thus I wanted to avoid pushing my app to Bluemix all the time, but instead wanted to test it locally first. Hence I downloaded and installed a local node.js environment, including the so-called node.js package manager "npm" first. npm allows you to install additional modules/code libraries. They are placed into the directory "node_modules". Within the program (or script), the modules are included and referenced via the "require" statement:

var express = require('express');

The above binds the "ExpressJS Web Application Framework for node". That framework is part of the node.js starter application on IBM Bluemix. With that basic application which is offered for download you can easily test whether the local installation work ok:

node app.js

The command which I executed in a regular shell launches the node.js runtime with the sample application. Based on the configuration it provides a small web application available on my laptop on port 3000. Accessing "http://127.0.0.1:3000" in my web browser shows the demo page. All ok.

To combine node.js and DB2 I require the DB2 database driver:

var ibmdb = require('ibm_db');

Just running the application again would return an error because the module has not been installed. Hence my next step in the command shell is:

npm install ibm_db

This invokes the node package manager and instructs it to download and install the IBM database client driver and related node.js API. Waiting for a minute it returned an error because it couldn't find the file "sqlcli1.h". This is an indicator that my local DB2 was missing the application development environment. Running "db2setup" again (as root), selecting "work with existing" and marking the application development package for installation solved the issue. After db2setup finished, I ran "npm install ibm_db" again and it was able to download, build and install that module.

To test my small app both locally and on Bluemix, I needed to obtain user and DB2 instance information for either the local environment or the Bluemix SQLDB service (DB2). This is done with the following code snippet (not that beauty as I just started...):

1:  // get DB2 SQLDB service information  
2:  function findKey(obj,lookup) {  
3:    for (var i in obj) {  
4:     if (typeof(obj[i])==="object") {  
5:       if (i.toUpperCase().indexOf(lookup) > -1) {  
6:        // Found the key  
7:        return i;  
8:       }  
9:       findKey(obj[i],lookup);  
10:     }  
11:    }  
12:    return -1;  
13:  }  
14:  var env = null;  
15:  var key = -1;  
16:  var db2creds=null;  
17:  if (process.env.VCAP_SERVICES) {  
18:     env = JSON.parse(process.env.VCAP_SERVICES);  
19:     key = findKey(env,'SQLDB');  
20:  }  
21:  if (!env) {  
22:    console.log("We are local");  
23:    var file = __dirname + '/db2cred.json';  
24:    try {  
25:     db2creds = require(file);  
26:    } catch(err) {  
27:     return {};  
28:    }  
29:  //  db2creds = JSON.parse(fileJSON);  
30:    console.log(db2creds);    
31:  } else {  
32:  var db2creds = env[key][0].credentials;  
33:    
34:  }  
35:  var connString = "DRIVER={DB2};DATABASE=" + db2creds.db + ";UID=" + db2creds.username + ";PWD=" + db2creds.password + ";HOSTNAME=" + db2creds.hostname + ";port=" + db2creds.port;  
36:    
37:  app.get('/db2', routes.db2test(ibmdb,connString));  
38:    
39:    

In the code I first search for the object with the Bluemix environment information. If it is not found the code assumes it is a local invocation. In that case the DB2 access information is loaded from the file "db2cred.json". It is a file I created in the application directory with a content like here:

Logo for my DB2 node.js app

{
    "hostname": "127.0.0.1",
    "host": "127.0.0.1",
    "port": 50000,
    "username": "hloeser",
    "password": "mytopsecretpassword",
    "db": "CLOUDDB"
}

The code uses the information about the hostname, port, and the user/password combination to create a connection string. That information together with the IBM Database Driver interface can be passed to a request handler in the node.js/Express runtime infrastructure (the "app.get()" call).

My small test application runs successfully both on my laptop as well as on IBM Bluemix. I plan to write more about it over the next days and to upload the code to my Github account. Bluemix-related posts can be accessed by this link.

Update: The follow-up article has been published here, showing geo IP lookup and logging into DB2.