Another learning experience, late ride

So yesterday was Chapter 2 in my SQL corruption trial by fire experience.  Early in the morning we saw the document imaging system generated an error importing a single file overnight.  When we looked in the Windows application log we saw the SQL server generated a 824 error which is bad news.  It basically means for some reason it was unable to write data.   I ran a dbcheck on the database and was not happy to see around 150 “inconsistencies” reported.  I was actually surprised the DB was still online and functioning as this was similar to the errors I saw when the DB went offline 10 days prior.

So I spent the rest of the day trying to determine my best course of action.  Contacting the application vendor last time proved to be pretty worthless but we placed a call with them anyway, which didnt get returned until later in the day, after I already decided on a new course of action. In my research I determined I was missing an important part of a good SQL management plan, regular dbchecks.  By running these checks you can catch inconsistencies early, making it easier to recover from them with minimal data loss.  The only time I ran these was after a problem already existed.

So it seemed like my best option was to pay for a Microsoft professional services incident where you pay $499 and they have a high level tech contact you to remedy the issue.  I placed the ticket around 3PM.  They had a four hour response window.  When I closed my door shortly before 5 I assumed we would be tackling the problem in depth the following day.  Well as I took a couple steps down the hall I heard my phone ring in the office.  I instantly wondered if it was Microsoft calling.  I ran back into my office and picked up the phone.  It was indeed Paul from SQL support group.

We embarked on a roughly three hour trouble shooting session that involved restoring back ups and testing their integrity.  We determined that even the back ups had the same inconsistencies as the production database.  The plan then shifted where the tech ran a repair on one of the backed up DB’s which fixed the inconsistencies by deleting the records that contained them.  The downside of this was a handful of data was deleted but in the big picture in a DB containing millions of images, losing a few is not really a concern.

My main concern was that the application still ran correctly after the clean up.  The tech swapped in the repaired DB and I brought the application back up.  I was able to test various aspects of the document imaging process, it all seemed to be working fine.  I thanked the tech for all of his help.  It was great to deal with someone so knowledgeable and even better that he was not from India.

I didn’t leave the office until a little before 8.  I felt badly because we made plans go riding the PTV’s with Katie and Daniel to look at more Christmas lights.  I was happy that even though I got home late they were still there and ready to ride.  Instead of going to Ave Maria as originally planned we went to the nearby Waterways development which had some very pretty lights as well.

It was a bit chilly with temps in the mid 60’s when combined with the wind from rolling around at 10+ mph.  Even so we had fun and were out there longer than we originally planned.  We made plans to meet up again Friday evening and head out to Ave Maria on the wheels, it should be a very festive and fun way to start off Christmas weekend.

I have known for awhile that the struts on the Prius have not been in good shape which probably helped accelerate the wear of the tires that were just replaced.  It sort of hammers over bumps.  I have now watched the replacement process for the front and rear struts/shocks.  It involved taking a LOT of stuff off just to access to the parts I want to replace.  So although it is involved I would not call it difficult work.  I think I will buy myself a Haynes repair manual shortly and tackle the job as an early 2017 challenge.

This will be my last working day until next Tuesday.  I am looking forward to a great four days with lots of giving peppered with some receiving.