Monday, August 3, 2009

Never under estimate a backout plan

Every well planned and thought out change could be implemented without problems in several environments. But it only takes small issue, a missed step or something that wasn't completely tested to cause an issue. Following a process to implement a change is important, but knowing what steps change be recovered from or rolled back are extremely important.
Can a step be repeated without an issue, what happens if you have an error after a step and the all dreaded forgetting a step? Checks through out the process and knowing if an error there means redoing everything or just running something to fix it at that spot will help prevent larger issues. Being able to isolate a change and know where the errors could come from will help solidify the change process and make a more robust implementation.
If this happens, then I have options to backout the change, and here are my steps to do that. If the change doesn't work or completely fails, I have a backup to restore and either start again, or live to try another day.
I could have applied this patch in 20 environments the exact same way, but run into issues where the code was different or parameters were slightly off, and it causes an issue, so how do I remove the patch, and what needs to be run afterwards to clean it up.
Compliance and IT processes should include test plans so you know what you need to test to validate the change as well as what you need to do to back out the change. Good backup strategies are also key here and understanding how long after the change the backups are still valid. Knowing how to put the database back to before the change would help if you have already hit that point of no return on the backups.
Implementing changes in databases can be a difficult process or it can be planned for the unexpected issues. Having test plans that hit the critical areas are important, and because of sizing and other factors, even the best test plans are not going to test everything all of the time. Being prepared that even if it is the last database for the change, something could go wrong and needing to revert the change might be inevitable. Steps created before the change, and then even testing that before applying the change in the all of the environments will elimate some of the fear of rolling out changes. Keeping the databases stable, available and productive after a change means good planning and being prepared in this area.

No comments:

Post a Comment