The process release doesn’t mean just deploying the latest artifacts to the production environment. Nowadays, pre-release and post-release activities are equally important as well. I am the technical leader of a tiny QA team here in Adform of 3 QA engineers including myself. I am blessed and honored to have Umair Mahmood and Abdul Hafeez into time who are not only dynamic test engineers but also two excellent human beings.
So coming back our production release. Our pre-release, release and post-release requires a few steps:
Some of the above steps are required for audit or just to fulfill the process. Some of them are very much important to share our release details with all other non-technical stakeholders across the organization. Also to ensure that the entire organization is aware of our release activity, we need to send out several updates in a particular slack channel. It helps stakeholders owning different other services to verify their changes in the pre-prod environment.
But where was the problem? Problem was that, initially, the above entire process used to take two complete man-days. two entire man-days! can you imagine! So we have to do something about this.
I crafted few stories out of the above process and proposed a tool to automate this entire process. So the tool which is a CLI developed and working as following flow:
So we decided to go ahead and translate the above flow diagram into a working tool. We have used the following technologies to achieve this:
- Slack Rest API – https://api.slack.com/
- Octokit (Git API) – https://octokit.github.io/rest.js/v18
- JIRA Rest API – https://www.npmjs.com/package/jira-connector
- Confluence Rest API – https://www.npmjs.com/package/confluence-api
As shown in the flow diagram above, the tool accepts three types of commands: prepare, deploy and switch. We are using blue-green deployment for our production environment and hence need to switch to the active color after the production deployment and verification. Also due to having Kubernetes and Docker images in place as production artifacts, made our life really easy to design and develop the tool.
Our SUT is an end result of multiple microservices. So we created the tool in such a way so that anyone in the organization should be able to use it by providing the necessary configuration (JSON format) and service name. I believe we are bound with the Intellectual property (IP) of the organization and hence it is not possible to make it open source yet.
What we have achieved
We managed to reduce our deployment time from two man-days down to few hours (2 hours max). Also, this entire effort is distributed with different commands as a result different steps of the deployment could be executed on demand.
A quick comparison
|Previous (Manual Process)||Now|
|Publish A release log in JIRA.||Run prepare command|
|Include links of stories in the release candidate to the release log document.|
|Collect all associated CI and PR links from Drone CI and Git and include them in the release log document.|
|Publish Release Note in Confluence with stories and PRs for all services of the release train.|
|Create release tags in Git for each of the services of the release train.|
|Send out communication with all necessary links (Release Log, Release Note, Pull requests, etc,.)|
|Deploy each service to the pre-prod and send out communication.||Run deploy command|
|Deploy to the production, switch to the active color and send out communication.||Run switch command|
This tool is producing the following deliverables for us:
- A well documented JIRA document as Release Log with following items:
- List and links of stories will be released from the current sprint (JIRA Story links)
- Pull request (PR) links from Git (with approval status)
- Drone CI links corresponding to the above PRs.
- A well documented Release Note (for non-technical stakeholders) in Confluence with following items:
- Stories organized in different sections based on the names of the services we are releasing.
- Links of all PRs associated with the stories.
- Release tags in Git (for each of the services we are releasing)
- We are also storing entire audit trail in slack for later use by posting deployment links along with deployment creator name
Well, the objective of this post was to express my gratitude to my teammates Abdul Hafeez and Umair Mahmood for their excellent and dedicated support during this journey. Especially Abdul Hafeez, who developed most of the tools. Also, Oak who is one of our BE developers but helped us with developing the entire slack communication mechanism during our busy time.
And I wanted to repeat myself again, automation is not only Selenium, ability to thinking out of the box to simplify our daily routine works by reducing manual works as much as possible is also one of the important attributes of automation. So keep thinking out of the box 🙂