Tag ix webhosting outage

So finally, finally, finally, my DB driven websites came back online early yesterday afternoon after being down since early Sunday morning.  Since I have had a presence on the internet in the 90’s, I have never had my content be down this amount of time.

To have this type of outage in 2014 is even more incredulous since high availability technology has improved so much in the last 5 years.  IX really has managed to set the bar to a new low.  I am pretty sure no major webhost in the last 5 years has had an outage of this duration.

The official description of the incident was IX’s RAID 14+2 SAN had a drive fail, and then as a hot spare was being swapped in, another drive failed which put the entire SAN into a failed state.

I have questions about why this would happen as the SAN was described as actually being 5 storage arrays, each one with 14 drives  in it.  I don’t follow why a failure of two drives in one of the 5 arrays would bring down the whole shebang.   If a massive array of 90 hard drives was toppled that easily then there was some poor engineering/planning/configuration utilized.

Last night I received an email from Faithi Said, the CEO of IX, declaring the recovery process was now officially complete with a promise of a full investigation of the circumstances that lead up to this mess.  Noticeably absent from this email was any mention of compensation for this clusterfck.  Many, many people literally lost thousands of dollars as a result of this outage.  A simple “sorry, our bad…” just isn’t going to cut it.

Last night on the way home I picked up a Craftsman tractor shade at the small Sears location in Bonita Springs.  I bought it online.  I had the option of having it shipped to a store and picking it up for free or paying almost $70 to have it shipped to my door.  It was a pretty easy decision to go pick it up.

The store appeared to have exactly one employee in it, at least while I was there.  It seemed pretty representative of the dire situation I read that the Sears family is in.  As I was waiting for the guy to be free I walked around the store, mostly looking at kitchen appliances since I have plans to update my set sometime in the future.  I was surprised at just how expensive most of what I looked at was.  There was a fridge sitting there tagged at $3600, wtf??  Sure it was big, something like 33 cubic feet but still it gave me sticker shot.  The floor was littered with high priced appliances, tons of them $2k or more.  Obviously Sears is not looking to be a low price leader, sheesh.

So the guy retrieves the box and asks me if I have the confirmation email on my phone.  I realized I did not.  I ordered the canopy the same day my IX servers went down.  The email confirmation was one of the casualties as initially IX f’d up and had things configured so people sending email to me would have gotten an undeliverable bounce message, meaning a number of emails sent to me in the first 24 hours vaporized in the the internet bit bucket, including the Sears email, great. Luckily he was able to confirm my order using some other method.

I actually have thought about getting a canopy for my yard tractors for years, ever since I saw my old neighbor Rick tooling around his yard on his sun shaded John Deere.  With my skin cancer history, avoiding unnecessary sun exposure is a smart move.

The addition of the canopy will make certain tractor maneuvers no longer possible under low hanging objects but I’ll trade that inconvenience for a less UV intensive and 10 degree cooler mowing experience.

This weekend we have a BIG movie to see, the sequel to 300 comes out.  As silly as it sounds, the original 300 really had a big impact on my life.  It was that movie and the published 300 workout that the actors on the film used that transformed my fitness routine.  Up until that point in my life my gym routines were pretty static, routine and yielded no real results beyond maintaining my current level of fitness.

The 300 workout is a brutal circuit that really pushed my mental and physical endurance boundaries.  I was REALLY into it.  I even have two pages dedicated to it the description of the workout and my log of 300 workouts which I maintained for 2-3 years.  When I did 300’s I had no real concept of the difference using full ROM and good form made.  Looking back now, with many of the movements I used poor form.

My 300 workouts eventually morphed into my current day calisthenics/bar focused exercise regimen that I have been doing for most of my 40’s.  So the point is, to me a new 300 movie is a big f’n deal. Cindy is a big 300 fan as well so we both are very excited to go see the film in 3D on Saturday night. I HOPE it lives up to epic-ness of the original.

I am planning a double endurance training weekend with hopefully a 5 mile run tomorrow backed up by a bike swim/brick on Sunday.  We also might be doing more work with getting the house ready for a full time second occupant.

For the last two days I have spent time in my underwear in front of two medical professionals.  Yesterday I had a physical required for my insurance coverage.  Because of my age the physical had the requirement of a DRE (digital rectal exam) aka. finger up the ass.  Thankfully my health advocate waived that requirement for me when I told her I just had a full man part exam late in 2013 as part of my visits to the urologist.

My physical went fine except for one thing, my blood pressure readings were on the mildly high side.  They actually took it twice, once at the start and again at the end of the visit.  The first numbers were like 140/76 and the second reading was 145/85.  I attribute the high numbers to slamming my customary large cup of coffee prior to the office visit and the involuntary anxiety any visit to a medical office invokes for me.

Despite these explanations, I was given a portable BP machine to check my readings the next two weeks just as a precaution. Both my mom and dad have high blood pressure, although I think theirs has as more to do with lifestyle choices.  When I checked it late afternoon the numbers had dropped to 123/80 so I feel confident coffee and nerves were the real catalyst.

My email services for my internet domains were restored yesterday, unfortunately my DB driven websites are still dead in the water.  They should hopefully be up and running within the next few hours.

The president of IX, Faithi Said, posted a more technical explantion of the hardware failure, claiming the array that failed utilized RAID 14+2, meaning it could tolerate a maximum of two simultaneous drive failures.  Supposedly one drive failed and then the drive that was actively being swapped in to replace it had a failure as well.

I am quite interested in seeing what sort of compensation will be offered to IX customers after this mess.  It seems surreal they have servers that are down for going on their 5th straight day.

Last night Cindy’s daughter stopped out with her Mustang.  The passenger side mirror had been broken for quite awhile.  She bought a replacement which I offered to put back on for her.  Replacing the mirror was extremely easy, little more than removing a trim piece and removing and replacing three screws.

While she was there I looked at some other cosmetic issues the car has like two damaged wheel wells and a missing front air dam.  Cindy’s daughter can be tough on things she owns, espcially vehicles and smart phones.  I think she has gone through 9 phones in the last three years. Afer doing the repair Katie and her friend hung out with us a bit which was nice, I don’t see much of her normally.

This morning I was once again in my boxers in  front of the dermatologist.  I got a clear screening for basal cell activity but got some cream for a rash that I have had for quite awhile.  Apparently I am not treating it long enough.  I have hit it with cream until it stops but then it will return.  The doc said I need to treat it a solid week or two beyond being symptomatic to make sure it is gone for good.

So I have been wearing my new Gunnar computer glasses all week at work.  As advertised they make things on my screens look cleaner, crisper and larger thanks to the slight magnification.  However there is something else going on, my eyes feel worse after wearing them all day.

One of the things that Gunnar users typically report is wearing the glasses eliminated the eye strain, dryness and headaches they were experiencing previously.  Although I don’t have headaches, my eyes feel MORE tired wearing the glasses then when I go at it naked.  They also feel dry, very odd.

A recurring message in reviews of the glasses is you need to give your eyes an adjustment period to get used to them so that is what I am planning to do.  I am just a little freaked out that I am developing symptoms that Gunnars are supposed to help correct.  If this is still the case a week or two later then I will have to reevaluate.

Here I am day 4 of no blog. IX webhosting is STILL in the process of recovering servers.  A four day outage is like years in internet time.  It will be quite interesting to see what sort of compensation plan they come up with for customers that haven’t already jumped ship. On a positive note, my email server was restored early this morning so at least that avenue of communication is once again flowing.

Over the weekend Cindy and I had a discussion about living arrangements.  The lease on her apartment is up soon.  The talk was about the idea of her moving in with me at the house.  To some it might sound like a big leap but for those that are more familiar with the situation it really isn’t that much of a stretch.  Cindy already spends a good chunk of time each week at my place.  We have spent plenty of time going through day in and day out together, enough to know there are no major personal obstacles/problems with living under the same roof .

Sharing the living expenses would be a major financial plus for both of us and Cindy is always eager to help me out at the homestead with whatever needs doing.  Really my only two concerns were not being able to my own thing and lack of space.

Cindy has never been the demanding type, just the opposite really.  She loves to do things with me but is fine doing her own thing if I want to do the same.  Space-wise she actually doesn’t have all that much stuff.  Getting her “moved in” really shouldn’t be that involved at all.

So anyway we decided to move ahead with the plan.  I think it will be a good thing for both of us.  It should be complete by this time next month.

So I arrived at work pretty early this morning, before hardly anyone else was there.  As I walked in the door I spotted some movement on the ledge in front of the windows.  As I focused I saw it was a small bird.  The bird looked wobbly and weak, having a hard time standing up and not moving much as I got closer to it, not a good sign.

I went inside but I quickly returned outside with a small box.  My office has about a half dozen feral cats that hang around, the little bird was sure to be killed by one of them if I let it be.  I wasn’t sure what I was accomplishing by bringing the bird inside, perhaps only giving it a quiet place to die.  It allowed me to pick it up and put it in the box with little fuss.

The bird was small and almost looked to me like it could be a baby mockingbird.  He didn’t seem to have a broken wing as he did move around a bit and was able to keep both of his wings tucked back in what looked like a normal position.

I wondered if it was a baby that dropped out of a nest.  I went back outside and looked at the nearby trees, seeing if I could spot a nest.  I saw none.

I got my hands on a plastic cup that I cut the sides off to create a makeshift water bowl for the bird.  I showed the bird to my boss.  She brought in some small pieces of bread in case it was hungry.  The bird didn’t show any interest in either although he did make some noises when I tried to move him around.  The noises sounded like something a baby bird would make.

I told Cindy about my office guest.  She said the wildlife conservancy located near the main post office would be the logical place for the bird to go, if it was a youngster hopefully they could feed it until it was ready to go back out in the wild.  I asked Cindy if she would mind picking it up since her work was nearby, she was fine with it.

So as I worked I would check in on my little feathered friend.  He seemed to just be resting but when he noticed me he seemed more alert than when I first found him.  I had the top of the box open since it seemed obvious he was not old enough to fly.

Cindy told me she was in the parking lot so I walked out with the bird in the box, showing some co-workers along the way.  I found Cindy in the lot and showed her the bird.  Just as she was saying how cute it was the bird took off and flew to a set of trees 100 feet away.  My mouth dropped open as I looked at Cindy.  He can fly!  I was surprised and relieved that the little guy apparently was ok.

I thanked Cindy for stopping by and went back inside with my now empty box.  I guess the bird did actually fly into the window and was just dazed/woozy.  I’m still glad I took him in as I am pretty sure one of the cats would have finished the little guy off without a second thought.

It’s always nice when an animal in need story has a happy ending.

Here I am going on my third day of internet darkness.   In a way I feel somewhat lucky that none of the sites I run/maintain are used for a real business.  When I read the comments from IX customers that are literally losing fistfuls of money and customers from this outage I feel badly for them.

This outage reenforces my dislike of MASSIVE consolidation of server resources, at least for things I manage personally.  Sure it’s really cool to have 10 servers shrunk down into one server, however if something just happens to go haywire with that one server, 10 servers go down with it.  That is why for mission critical things in our environment I use a traditional server model with individual RAID protected storage.  Call me old fashioned but I just don’t feel great having too many eggs in one basket.  I like my castrophic failures to be contained in nice little pockets of chaos instead of network wide.

The status blog for IX Webhosting is lit up with over 1500 comments from customers that are besides themselves about the slow recovery from this failure.  I find myself chuckling at the pleas to DO MY SERVER FIRST!  This is probably the longest outage I have had of my internet presence since it was implemented back in the 90’s.   Yes it isn’t good to be cut off from email for this long since it is my most heavily used and preferred vehicle of communication, but there is literally nothing I can do about it at this point.