Back from the dead, come full circle
So finally, finally, finally, my DB driven websites came back online early yesterday afternoon after being down since early Sunday morning. Since I have had a presence on the internet in the 90’s, I have never had my content be down this amount of time.
To have this type of outage in 2014 is even more incredulous since high availability technology has improved so much in the last 5 years. IX really has managed to set the bar to a new low. I am pretty sure no major webhost in the last 5 years has had an outage of this duration.
The official description of the incident was IX’s RAID 14+2 SAN had a drive fail, and then as a hot spare was being swapped in, another drive failed which put the entire SAN into a failed state.
I have questions about why this would happen as the SAN was described as actually being 5 storage arrays, each one with 14 drives in it. I don’t follow why a failure of two drives in one of the 5 arrays would bring down the whole shebang. If a massive array of 90 hard drives was toppled that easily then there was some poor engineering/planning/configuration utilized.
Last night I received an email from Faithi Said, the CEO of IX, declaring the recovery process was now officially complete with a promise of a full investigation of the circumstances that lead up to this mess. Noticeably absent from this email was any mention of compensation for this clusterfck. Many, many people literally lost thousands of dollars as a result of this outage. A simple “sorry, our bad…” just isn’t going to cut it.
Last night on the way home I picked up a Craftsman tractor shade at the small Sears location in Bonita Springs. I bought it online. I had the option of having it shipped to a store and picking it up for free or paying almost $70 to have it shipped to my door. It was a pretty easy decision to go pick it up.
The store appeared to have exactly one employee in it, at least while I was there. It seemed pretty representative of the dire situation I read that the Sears family is in. As I was waiting for the guy to be free I walked around the store, mostly looking at kitchen appliances since I have plans to update my set sometime in the future. I was surprised at just how expensive most of what I looked at was. There was a fridge sitting there tagged at $3600, wtf?? Sure it was big, something like 33 cubic feet but still it gave me sticker shot. The floor was littered with high priced appliances, tons of them $2k or more. Obviously Sears is not looking to be a low price leader, sheesh.
So the guy retrieves the box and asks me if I have the confirmation email on my phone. I realized I did not. I ordered the canopy the same day my IX servers went down. The email confirmation was one of the casualties as initially IX f’d up and had things configured so people sending email to me would have gotten an undeliverable bounce message, meaning a number of emails sent to me in the first 24 hours vaporized in the the internet bit bucket, including the Sears email, great. Luckily he was able to confirm my order using some other method.
I actually have thought about getting a canopy for my yard tractors for years, ever since I saw my old neighbor Rick tooling around his yard on his sun shaded John Deere. With my skin cancer history, avoiding unnecessary sun exposure is a smart move.
The addition of the canopy will make certain tractor maneuvers no longer possible under low hanging objects but I’ll trade that inconvenience for a less UV intensive and 10 degree cooler mowing experience.
This weekend we have a BIG movie to see, the sequel to 300 comes out. As silly as it sounds, the original 300 really had a big impact on my life. It was that movie and the published 300 workout that the actors on the film used that transformed my fitness routine. Up until that point in my life my gym routines were pretty static, routine and yielded no real results beyond maintaining my current level of fitness.
The 300 workout is a brutal circuit that really pushed my mental and physical endurance boundaries. I was REALLY into it. I even have two pages dedicated to it the description of the workout and my log of 300 workouts which I maintained for 2-3 years. When I did 300’s I had no real concept of the difference using full ROM and good form made. Looking back now, with many of the movements I used poor form.
My 300 workouts eventually morphed into my current day calisthenics/bar focused exercise regimen that I have been doing for most of my 40’s. So the point is, to me a new 300 movie is a big f’n deal. Cindy is a big 300 fan as well so we both are very excited to go see the film in 3D on Saturday night. I HOPE it lives up to epic-ness of the original.
I am planning a double endurance training weekend with hopefully a 5 mile run tomorrow backed up by a bike swim/brick on Sunday. We also might be doing more work with getting the house ready for a full time second occupant.