Friday, June 27, 2014

Some fun with Bluemix, Cloud Foundry, Python, JSON, and the Weather

Bluemix: Bring Your Own Community
Today, I wanted to try out running a Python application in IBM Bluemix. And I wanted to check out weather reports for some possible weekend destinations. Why not combine it?

Because IBM Bluemix is based on Cloud Foundry, I am using the Cloud Foundry-related tools (mainly "cf") to interact with Bluemix. A first challenge is that a Python runtime is not built into Bluemix/Cloud Foundry and I have to tell it to prepare the runtime for me. It is called "Bring Your Own Community". When you click that icon, some helpful description comes up, pointing you to how to get started.
Bluemix: Get started with own buildpacks

The trick is use some scripting to package framework and runtime support and it is called buildpack. For Python there are several buildpacks available and I am going to use this one. It is specified whenever pushing the application from my local machine to Bluemix:

cf push weatherFN -b https://github.com/ephoning/heroku-buildpack-python.git

As a starter, I downloaded a sample Python-based Web application that is deployable to Bluemix/Cloud Foundry. It comes with 5 files, one of them a small "hello.py", the actual application. The other files are:
  • runtime.txt - it specifies the Python version
  • requirements.txt to list the dependencies on other Python modules
  • Procfile - tells the runtime what and how to start
  • README.md with some basic introductions
With the starter snippet ready, I looked for a source for weather data. Data for airports worldwide is available, e.g., at the US National Weather Service. I could fetch the current weather for Friedrichshafen airport like this encoded as METAR:

2014/06/27 11:20
EDNY 271120Z 22007KT 180V240 CAVOK 24/06 Q1017

However, it would require decoding (for untrained people like myself). So I looked further and I found OpenWeatherMap. They offer the weather data in various formats, including JSON. The data for Friedrichshafen can easily be fetched like this and look like in this sample:

{"coord":{"lon":9.48,"lat":47.65},"sys":{"message":0.0037,"country":"DE","sunrise":1403839575,"sunset":1403897051}, "weather":[{"id":800,"main":"Clear","description":"Sky is Clear","icon":"01d"}], "base":"cmc stations","main":{"temp":297.46,"humidity":41,"pressure":1010.562,"temp_min":297.15,"temp_max":297.95}, "wind":{"speed":1,"gust":4,"deg":61}, "clouds":{"all":0},"dt":1403869524,"id":2924585,"name":"Friedrichshafen","cod":200}

Python has a built-in JSON module and it is easy to load and dump the data:
    wdata = json.load(urllib.urlopen(url))
    print json.dumps(wdata, indent=2)


A single field can be accessed using array notation (description of current weather condition):
  wdata["weather"][0]["description"]

After some local testing I pushed the code to Bluemix as shown earlier. My Web app is called "weatherFN" as in "weather in Friedrichshafen". When the index page is requested the app redirects to the weather for Friedrichshafen:
http://weatherfn.mybluemix.net/weather/Friedrichshafen
Weather in Friedrichshafen
The same can be done for London or Recife (Brazil), the soccer match USA vs. Germany was held there yesterday:
Weather in London
Weather in Recife

Weather looks nice, I learned more about IBM Bluemix and how to use Python today. Now it is almost time for the weekend.

Last but not least, here is the entire Python code that I used:
 import os  
 from flask import Flask,redirect  
 import urllib  
 import json  
 BASE_URL = "http://api.openweathermap.org/data/2.5/weather?q="  
 app = Flask(__name__)  
 @app.route('/')  
 def index():  
   return redirect('/weather/Friedrichshafen')  
 @app.route('/weather/<city>')  
 def weather(city):  
   url = "%s/%s" % (BASE_URL, city)  
   wdata = json.load(urllib.urlopen(url))  
   print json.dumps(wdata, indent=2)  
   page='<title>current weather for '+wdata["name"]+'</title>'  
   page +='<h1>Current weather for '+wdata["name"]+' ('+wdata["sys"]["country"]+')</h1>'  
   page += '<br/>Min Temp. '+str(wdata["main"]["temp_min"]-273.15)+'<br/>'  
   page += '<br/>Max Temp. '+str(wdata["main"]["temp_max"]-273.15)+'<br/>'  
   page += '<br/>Current Temp. '+str(wdata["main"]["temp"]-273.15)+'<br/>'  
   page += '<br/>Weather: '+wdata["weather"][0]["description"]+'<br/>'  
   return page  
 port = os.getenv('VCAP_APP_PORT', '5000')  
 if __name__ == "__main__":  
      app.run(host='0.0.0.0', port=int(port))  

There are follow-up articles to this one describing how I enabled a custom domain for my Bluemix application and how I added a Cloudant / couchDB to my Python application on Bluemix.

Tuesday, June 24, 2014

Why we need and have workload management

Wikipedia
While working in your office a rare visitor from another location stops by. Time for a break to connect on the latest gossip, but not too long. On the way back to your office your boss asks you to call someone from the client team to clarify some technical issues and you have to squeeze it in between two important customer calls. And you just received a text message that your wife cannot pick up the kids and you need to leave on time this afternoon to do it. Workload Management (WLM) in real life. Everybody seems to be doing WLM, some better, some not so well. And there are many unwritten rules.

In a database system like DB2 there is also a built-in Workload Management. If you are using BLU Acceleration, it is activated by default and some rules have been defined, else it is switched off. Why turn it on and use it? Same reasons as in real life:
  • A "fair" allocation of time and resources between different work items/applications is needed ("work / life balancing"?).
  • Response time for critical tasks or some type of work is important and needs to be protected against less important tasks ("your mother-in-law visits, take care of her").
  • Implementation of rules to control and regulate the system behavior ("kids in bed by 8pm means time for you to watch soccer").
  • Deal with rogue queries that threaten regular operations ("kids bring over the entire neighborhood").
  • The system (sometimes) is overloaded and you have to set priorities ("no party this weekend").
All this can be done with the DB2 Workload Manager. It allows to identify different types of activities (work), manage them based on rules that govern available resources and set controls, and to monitor the system behavior. The database workload manager can be integrated with the operating system (OS) workload manager on AIX and Linux. This is especially useful when more than a single database is active and resources need to be controlled on a higher level ("sync your family calendar with the grandparents").

Does Workload Management help? Yes, it does. However, similar to family life it is possible that because of resource shortage not all planned tasks can be performed. Maybe time for an upgrade ("hire some help, do not get more kids... :)").

I plan to discuss DB2 WLM details in future articles, workload permitting...

Wednesday, June 18, 2014

Video: Introduction to IBM Bluemix User Interface

Currently, I am taking a look at IBM's new Platform-as-as-Service (PaasS) offering code-named "Bluemix". Right now it is in open beta and I signed up on its official website http://bluemix.net. Bluemix is based on the open source Cloud Foundry. But how do you get started and get a good introduction to what is offered?

I read the article "What is IBM Bluemix?" on IBM developerWorks. It gives some background information and details that Bluemix offers Development Frameworks (develop in, e.g., Java, Ruby, or Node.js), Application Services (DB2, MongoDB, MySQL, and others), and Clouds, i.e., you can deploy your applications (apps) to public, private, or other clouds.

Next, I watched the following video which gives a nice overview about the different elements of the Bluemix user interface, including the dashboard.




Equipped with that background information and knowing about the UI, the next stop is the Bluemix website offering articles and sample code. One of the examples is on how to build a simple business intelligence (BI) service using Ruby and based on DB2 with BLU Acceleration, code samples included.

If you haven't tried it yet, you can still sign up for the free beta here: http://bluemix.net. Enjoy!

Tuesday, June 17, 2014

DB2 Screenshot Quiz: Where is this taken from?

I am using different DB2-related services, such as the new Knowledge Center for DB2, BLU for Cloud (DB2 with BLU Acceleration in the Cloud), IBM Bluemix, and of course a local DB2 installation. Where did I find the following graphic? It is part of one of the above mentioned services...






Let me know by comment or direct email.

Monday, June 2, 2014

Improved db2look in DB2 to mimic database environments

Some of the advertised improvements in the recent DB2 10.1 Fixpack 4 apply to the long existing tool db2look. Two new options have been added: "-createdb" and "-printdbcfg". The first is used to generates the CREATE DATABASE command and its options, the second to generate statements to reapply the database configuration.

As it is new, I wanted to test it myself. First I created a database "lt" (as in "Look Test") with non-standard options. Next was to invoke db2look:

db2look -d lt -createdb -printdbcfg -o lt.out
-- No userid was specified, db2look tries to use Environment variable USER
-- USER is: HLOESER
-- Output is sent to file: lt.out
-- Binding package automatically ...
-- Bind is successful
-- Binding package automatically ...
-- Bind is successful


The generated output file starts with the usual environment and version information, then follows the section to recreate the database:

--------------------------------------------------------
-- Generate CREATE DATABASE command
--------------------------------------------------------

CREATE DATABASE LT
        AUTOMATIC STORAGE NO
        USING CODESET ISO8859-1 TERRITORY de
        COLLATE USING IDENTITY
        PAGESIZE 8192
        DFT_EXTENT_SZ 32


...

;

As you can see, I didn't use automatic storage, used a local, non-Unicode codepage and German territory, an identity collation and 8 kByte pages. Thereafter follow the parameters for the catalog, temporary, and user  tablespaces (not shown). After the database creation is completed, the next is the CONNECT statement:

CONNECT TO LT;



Once the database connection is established, another new section starts. It reapplies the database configuration:

--------------------------------------------------------
-- Generate UPDATE DB CFG commands
--------------------------------------------------------

-- The db2look command generates the UPDATE DB CFG statements
-- to replicate the database configuration parameters based on
-- the current values in the source database.
-- For the configuration parameters which support AUTOMATIC,
-- you need to add AUTOMATIC to the end
-- if you want the DB2 database to automatically adjust them.

--UPDATE DB CFG FOR LT USING ALT_COLLATE      ;

UPDATE DB CFG FOR LT USING STMT_CONC          OFF ;

UPDATE DB CFG FOR LT USING DISCOVER_DB        ENABLE ;

UPDATE DB CFG FOR LT USING DFT_QUERYOPT       5 ;
UPDATE DB CFG FOR LT USING DFT_DEGREE         1 ;

...



Right now the enhancements are only available in the just recently released fixpack of DB2 10.1. As with other improvements, I would expect it to be available for the newer DB2 10.5 release soon.