It’s not the best time of week, when after long hours of preparation and intense work during a deployment window, your change fails. Besides, it also breaks other things. Sometimes it is evident immediately after deployment, but sometimes it is discovered hours or days later. If you’re lucky, you can apply an emergency change to fix a fairly apparent root cause – for example, one of the items is missing from the deployment checklist. However, in many cases this will not be possible and you will need to reverse the change.
A key to a successful withdrawal is having a plan. Yes, a withdrawal plan, which is often overlooked. After all, you want your withdrawal to be an honorable surrender, not a panic escape. To limit the damage to the company and its reputation, you must remain in control of the situation. To do that, the engineering team needs to know what to do and the service desk needs to keep the company informed.
A withdrawal plan aims to keep you in control. It’s your insurance policy against Murphy’s Law. Let’s be honest with ourselves: we don’t insure everything. Don’t prepare a formal retirement plan for every change. Just make sure the team can verbally describe how to get back in case things get tough.
However, you need a more formal plan for more complex changes. The creation of such a plan is one of the least favorable activities of many technicians. That is why the change manager should be responsible for doing it. It should be included with the rest of the change documentation, ready to use if necessary.
A good withdrawal plan should include:
- low-level technical instructions,
- specific communication instructions, with contact names.
The list of technical instructions is created by reversing the order of the activities in your implementation plan and describing how to go back through each of the steps performed. It can be relatively straightforward if most of the work could be accomplished by restoring the most recent backup. Consider a sample recall plan for such a scenario:
- Notify the Service Desk about the start of the termination plan. (Call them, send an email or generate a fine, state specifically).
- Disable user access to the system. (How? List the actions).
- Restore the backup made before the change was implemented. (List the necessary actions).
- Perform system health checks (list them all).
- Enable user access.
- Notify the service desk of successful cancellation.
Often times the plan will be more complex than it appears. There can be many more restoration steps, involving multiple databases, file systems, and other areas of the IT infrastructure. The basic template still applies. It must be detailed and adapted to each organization and each change. It goes without saying that every share must have an owner, so make sure it’s clear who does what.
Communication with the Service Desk is very important. Communication in general should be part of the plan to maintain control over the situation. Additionally, the business needs to know that IT is in control. The Service Desk must be in charge of projecting the image of control towards the business. They can do this by issuing regular communications if the business impact is severe enough. They will also receive calls from dissatisfied users and inform them about the status of the resolution.
A cancellation plan is your insurance policy. It is up to you to have it or not. It is recommended to have it for every complex change, because business continuity and IT credibility are at stake. Start by preparing such a plan for the more complex change ahead in your process. Then build on that and in time you’ll have it ready for all the high-stakes changes.