Miserable Monday

Yesterday was one of those text book miserable IT Mondays where something breaks over the weekend and your first task of the day is to fix it.  Well this was a rather huge break, the database of our document imaging server became corrupted somehow on Friday evening, before any back ups had run of course.

I spent the first half of the day working with the vendor for the imaging system.  It became apparent very quickly this guy really had no idea what to do.  I was clinging to repairing the DB and bringing it back online to eliminate any data loss.   When it was apparent this guy was just spinning his wheels I moved on to plan two, database restore from our back up appliance.

We had a full database back up from Tuesday night and a differential backup from Thursday so if all went as planned I could at least only lose one day of work.  Well of course that plan didn’t work out either.  For some reason, unknown to the back up vendor as well, the differential restore would fail at the end, leaving the DB stuck in a “Restoring” state.  However the full back up from Tuesday restored fine.  I went through at least a half dozen attempts of restoring the massive 90 gig DB until I had to make another command decision.

It was now 3pm.  If I just restored Tuesdays back up the system would be back online and allow the clerks to process their daily work they have been sitting on.  However I was then going to be losing three days of processing from last week.  I decided it was more important to get the wheels turning again and restored Tuesdays back up.

The way this system works is once a document is pulled into the system it is converted to a pdf which is placed in a folder broken down logically by date.  So any work from those three days was already sitting out there in pdf form, the problem is it no longer was linked to the database.  The rather klutzy fix is to re-import these pdfs into the system manually, a pain in the ass but doable.

I stated the process with my asst net admin late in the day, staying until a little after 5:30.  I did some more work from home last night and was into the office early this morning to continue the process.  For an IT job, things are abnormally stable with my current environment compared to what I was used to in previous jobs.  However days like yesterday were a reminder of the feet to the fire environment that can always be lurking around any corner.