Use a Delivery Pipeline to rotate credentials

Job in Delivery Pipeline to rotate keys

In my recent posts I touched on updating credentials for solutions deployed on IBM Cloud Functions or using Cloud Foundry on IBM Cloud. Today, I am showing you how to rotate API keys and passwords for a containerized solution on IBM Kubernetes Service that makes use of a delivery pipeline (devops). I am going to use the app discussed in the tutorial on how to apply end to end security to a cloud application.

Overview

The app discussed in the tutorial provides a solution for securely share files. It uses Cloud Objects Storage and a NoSQL database to manage files and related metadata. Access is controlled via IBM Cloud AppID which provides authentication and identity services. The code for the app and a toolchain to automatically create required services and keys as well as to deploy the app to a Kubernetes are provided in the GitHub repository secure-file-storage.

Rotating credentials

Security works best when it is easy to integrate. Thus, I looked into adding stages to the delivery pipeline for rotating credentials. A quick assessment showed that credentials are used in three different ways:

1. The app itself access Cloud Object Storage and a Cloudant database. Both credentials are made available in a single Kubernetes secret. A requirement is to keep the app online while changing the secret, i.e., a zero-downtime update.
2. AppID is deployed as cluster integration, the cluster ingress (network access) requires a valid authentication. The access protection needs to be kept active all the time. A zero-downtime update is required, too.
3. During the container deployment, the deploy process uses an image pull secret to access the IBM Cloud container registry.

For all three scenarios, the steps to rotate the access data involves renaming the existing access key, creating new credentials with the previous name, applying the new credentials to the Kubernetes secret (updating or recreating it), and eventually to deactivate the old credentials by deleting the old service key. To handle each of the three scenarios independently, I created three scripts (look for pipeline-ROTATE_XXX_CREDENTIALS.sh).

There are different ways of how the secrets are composed and how they are updated. You can read the source for details, but I want to point out few interesting things I learned. The first is that you can only create or delete secrets, but there is a still a neat way of updating them. It involves creating a new secret in dry run mode, then piping it to apply to replace the existing content:

kubectl create secret secret-type --from-source-type= --dry-run=true -o yaml | kubectl apply -f -

The second detail is a new kubectl rollout subcommand to restart apps with zero downtime:

kubectl rollout restart deployment-name

It is used by the script to rotate the storage credentials. The command first brings up new pods with the app which has the new credentials, then it shuts down the old (existing) pods and the older credentials are no longer used.

Pipeline integration

The three scripts to rotate the credentials can be easily integrated into a delivery pipeline as manual stages. That way, they can access the cluster and account information that was used to build and deploy the app. With a single click and no further configuration I am able to update passwords and API keys when it is required, either periodically or when a team member leaves or if there is a (suspected) incident:

Rotate credentials with a click

Easy, right? Check out the updated tutorial on how to apply end to end security to a cloud application and the related GitHub repository.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.