Tag ix webhosting crash

For the last two days I have spent time in my underwear in front of two medical professionals.  Yesterday I had a physical required for my insurance coverage.  Because of my age the physical had the requirement of a DRE (digital rectal exam) aka. finger up the ass.  Thankfully my health advocate waived that requirement for me when I told her I just had a full man part exam late in 2013 as part of my visits to the urologist.

My physical went fine except for one thing, my blood pressure readings were on the mildly high side.  They actually took it twice, once at the start and again at the end of the visit.  The first numbers were like 140/76 and the second reading was 145/85.  I attribute the high numbers to slamming my customary large cup of coffee prior to the office visit and the involuntary anxiety any visit to a medical office invokes for me.

Despite these explanations, I was given a portable BP machine to check my readings the next two weeks just as a precaution. Both my mom and dad have high blood pressure, although I think theirs has as more to do with lifestyle choices.  When I checked it late afternoon the numbers had dropped to 123/80 so I feel confident coffee and nerves were the real catalyst.

My email services for my internet domains were restored yesterday, unfortunately my DB driven websites are still dead in the water.  They should hopefully be up and running within the next few hours.

The president of IX, Faithi Said, posted a more technical explantion of the hardware failure, claiming the array that failed utilized RAID 14+2, meaning it could tolerate a maximum of two simultaneous drive failures.  Supposedly one drive failed and then the drive that was actively being swapped in to replace it had a failure as well.

I am quite interested in seeing what sort of compensation will be offered to IX customers after this mess.  It seems surreal they have servers that are down for going on their 5th straight day.

Last night Cindy’s daughter stopped out with her Mustang.  The passenger side mirror had been broken for quite awhile.  She bought a replacement which I offered to put back on for her.  Replacing the mirror was extremely easy, little more than removing a trim piece and removing and replacing three screws.

While she was there I looked at some other cosmetic issues the car has like two damaged wheel wells and a missing front air dam.  Cindy’s daughter can be tough on things she owns, espcially vehicles and smart phones.  I think she has gone through 9 phones in the last three years. Afer doing the repair Katie and her friend hung out with us a bit which was nice, I don’t see much of her normally.

This morning I was once again in my boxers in  front of the dermatologist.  I got a clear screening for basal cell activity but got some cream for a rash that I have had for quite awhile.  Apparently I am not treating it long enough.  I have hit it with cream until it stops but then it will return.  The doc said I need to treat it a solid week or two beyond being symptomatic to make sure it is gone for good.

So I have been wearing my new Gunnar computer glasses all week at work.  As advertised they make things on my screens look cleaner, crisper and larger thanks to the slight magnification.  However there is something else going on, my eyes feel worse after wearing them all day.

One of the things that Gunnar users typically report is wearing the glasses eliminated the eye strain, dryness and headaches they were experiencing previously.  Although I don’t have headaches, my eyes feel MORE tired wearing the glasses then when I go at it naked.  They also feel dry, very odd.

A recurring message in reviews of the glasses is you need to give your eyes an adjustment period to get used to them so that is what I am planning to do.  I am just a little freaked out that I am developing symptoms that Gunnars are supposed to help correct.  If this is still the case a week or two later then I will have to reevaluate.

Here I am day 4 of no blog. IX webhosting is STILL in the process of recovering servers.  A four day outage is like years in internet time.  It will be quite interesting to see what sort of compensation plan they come up with for customers that haven’t already jumped ship. On a positive note, my email server was restored early this morning so at least that avenue of communication is once again flowing.

Over the weekend Cindy and I had a discussion about living arrangements.  The lease on her apartment is up soon.  The talk was about the idea of her moving in with me at the house.  To some it might sound like a big leap but for those that are more familiar with the situation it really isn’t that much of a stretch.  Cindy already spends a good chunk of time each week at my place.  We have spent plenty of time going through day in and day out together, enough to know there are no major personal obstacles/problems with living under the same roof .

Sharing the living expenses would be a major financial plus for both of us and Cindy is always eager to help me out at the homestead with whatever needs doing.  Really my only two concerns were not being able to my own thing and lack of space.

Cindy has never been the demanding type, just the opposite really.  She loves to do things with me but is fine doing her own thing if I want to do the same.  Space-wise she actually doesn’t have all that much stuff.  Getting her “moved in” really shouldn’t be that involved at all.

So anyway we decided to move ahead with the plan.  I think it will be a good thing for both of us.  It should be complete by this time next month.

So I arrived at work pretty early this morning, before hardly anyone else was there.  As I walked in the door I spotted some movement on the ledge in front of the windows.  As I focused I saw it was a small bird.  The bird looked wobbly and weak, having a hard time standing up and not moving much as I got closer to it, not a good sign.

I went inside but I quickly returned outside with a small box.  My office has about a half dozen feral cats that hang around, the little bird was sure to be killed by one of them if I let it be.  I wasn’t sure what I was accomplishing by bringing the bird inside, perhaps only giving it a quiet place to die.  It allowed me to pick it up and put it in the box with little fuss.

The bird was small and almost looked to me like it could be a baby mockingbird.  He didn’t seem to have a broken wing as he did move around a bit and was able to keep both of his wings tucked back in what looked like a normal position.

I wondered if it was a baby that dropped out of a nest.  I went back outside and looked at the nearby trees, seeing if I could spot a nest.  I saw none.

I got my hands on a plastic cup that I cut the sides off to create a makeshift water bowl for the bird.  I showed the bird to my boss.  She brought in some small pieces of bread in case it was hungry.  The bird didn’t show any interest in either although he did make some noises when I tried to move him around.  The noises sounded like something a baby bird would make.

I told Cindy about my office guest.  She said the wildlife conservancy located near the main post office would be the logical place for the bird to go, if it was a youngster hopefully they could feed it until it was ready to go back out in the wild.  I asked Cindy if she would mind picking it up since her work was nearby, she was fine with it.

So as I worked I would check in on my little feathered friend.  He seemed to just be resting but when he noticed me he seemed more alert than when I first found him.  I had the top of the box open since it seemed obvious he was not old enough to fly.

Cindy told me she was in the parking lot so I walked out with the bird in the box, showing some co-workers along the way.  I found Cindy in the lot and showed her the bird.  Just as she was saying how cute it was the bird took off and flew to a set of trees 100 feet away.  My mouth dropped open as I looked at Cindy.  He can fly!  I was surprised and relieved that the little guy apparently was ok.

I thanked Cindy for stopping by and went back inside with my now empty box.  I guess the bird did actually fly into the window and was just dazed/woozy.  I’m still glad I took him in as I am pretty sure one of the cats would have finished the little guy off without a second thought.

It’s always nice when an animal in need story has a happy ending.

Here I am going on my third day of internet darkness.   In a way I feel somewhat lucky that none of the sites I run/maintain are used for a real business.  When I read the comments from IX customers that are literally losing fistfuls of money and customers from this outage I feel badly for them.

This outage reenforces my dislike of MASSIVE consolidation of server resources, at least for things I manage personally.  Sure it’s really cool to have 10 servers shrunk down into one server, however if something just happens to go haywire with that one server, 10 servers go down with it.  That is why for mission critical things in our environment I use a traditional server model with individual RAID protected storage.  Call me old fashioned but I just don’t feel great having too many eggs in one basket.  I like my castrophic failures to be contained in nice little pockets of chaos instead of network wide.

The status blog for IX Webhosting is lit up with over 1500 comments from customers that are besides themselves about the slow recovery from this failure.  I find myself chuckling at the pleas to DO MY SERVER FIRST!  This is probably the longest outage I have had of my internet presence since it was implemented back in the 90’s.   Yes it isn’t good to be cut off from email for this long since it is my most heavily used and preferred vehicle of communication, but there is literally nothing I can do about it at this point.

Well this is an unusual occurrence, doing a blog posting in Wordpad.  I will be outlining the reason for this later in the entry.

I got home Friday night and noticed the pool cage rescreening job had been completed.  My eyes were immediately drawn to the two corners of the cage which had small wrinkles in the screen.  Other than that the rest of the screen looked really good.

We had a beautiful morning for the race I timed Saturday morning.  The turnout for the event was very small with somewhere around 175 people signing up.  This race takes place on the same weekend as a large half marathon in Fort Myers so it is set up for failure.  It really needs to be moved to some other weekend in the future.

The race was so small that I was able to shoot some aerial video with the Phantom as finishers streamed across the finsih line.  Both spectators and participants were fascinated by the buzzing, 4 bladed camera platform above.  The video I captured turned out pretty well.  I hope to be able to do this more often in the future.

On Saturday afternoon after finishing up post race work I went to pick up the dogs.  I told Ali I could keep them overnight.  It worked out well since she was one of those people running the Fort Myers race on Sunday.

Cindy and I decided to spend Saturday night at the house instead of hitting the movies.  I had a fresh Netflix rental to watch anyway, Rush.  Rush has the guy that plays Thor in it but if you ask me, the story, at least the more compelling part of the story is about Nikki Lauda, Thor’s nemesis in Formula 1 racing.  I remember hearing Nikki’s name growing up during ABC Wide World of Sport broadcasts but I had no idea he went through the hell depicted in the film.

I found the movie to be good, quite surprising since I could care less about racing in general.  I’d give it a strong B+

Sunday morning started off slow, it was very damp and foggy.  We scrapped another biking/swimming brick for another run around the track instead.  I was surprised that simply moving from lane 1 to lane 2 on the track added nearly a 1/4 mile to the total distance according to my GPS.  My Garmin said we covered 3.37 miles at roughly an 8:30 pace doing 12 laps around the oval.

The girls had fun hanging out at the house as usual.  They just love being able to walk around outside without any major obstacles.

The pool screen guy stopped by Sunday afternoon to collect the balance of the payment.  I pointed out the screen wrinkles.  He said he was hoping a day or two in the sun would have smooth them out. (really?)  He said he would be back on Monday to take care of them.

While he was there I asked him  about the technique used in rescreening an entire cage.  Since I have spent many irritating hours replacing screen panels I was curious to hear how you do it on a large scale.  He said they attach one end and then unroll the screen to cover an entire row of panels, they pull it tight, drop in the spline and then cut the screen.  It saves a whole bunch of cutting and measuring.

On Sunday evening Cindy and I watched Inequality for All, a film produced by Robert Reich, a name/face you may recognize.  Robert’s most famous position was as labor secretary under Bill Clinton. In recent years he has been popping up all over the place making commentary on the growing wealth disparity in the United States, both why it has been accelerating the last 30 years and why it is a huge problem.

The documentary, which is available for streaming free on Netflix was 90 minutes of some stuff I already knew and a lot of stuff I didn’t.  The start of this canyon size gap of wealth disparity started pretty clearly in the late 1970’s when the wages for the majority of the population stopped following the same general upward curve as the economy.  Wages flat lined while the economy continued to generate more and more revenue.  The overwhelming percentage of that additional revenue went directly into the pockets of an amazingly small amount of individuals.

The film will do a much better job of explaining the issue than I will regurgitating the facts here but let me simply say that any logical human being should see why this trend is unsustainable and self destructive in the long term, unless the goal of our country is to exist in a feudal society.

Once the ultra-rich cross a certain point of control in politics, reversing that trend becomes a task of monumental portions.  How do you change policies that benefit massive corporations if the massive corporations have bought enough influence in Washington to ensure nothing gets done?

Despite Roger’s small stature, I don’t think he is even 5 feet tall, he has a big voice on this subject.  I have followed him on Facebook for quite awhile and generally find myself shaking my head in agreement to most of what he says.

He does have some ideas that seem a bit far out there, even for me, like wanting to raise the minimum wage nearly 50% beyond the new levels Obama had mentioned ($10 and hour).  Robert wants $15 an hour as the new minimum hourly wage.

Robert says without a strong middle class, the country will rot from within, as it is the middle class that does the majority of the spending that drives consumption of products and services in our country.  His thinking is if people make a higher minimum wage, more of that spending can occur.  Of course there are counterpoints to this theory such as increasing the minimum wage could cause even greater unemployment or drive up inflation.

I did find it quite interesting that the one woman featured in the documentary was a Costco employee,  a big chain that sells goods warehouse style like Wal-mart does with Sam’s Club.  The retail market for the most part has been operating under the guise that there is absolutely no way that a retailer can offer employees decent wages and still be profitable.  Well Costco is a huge middle finger to that concept.  The woman was making $21+ an hour.  It can be done.

Please take an hour and a half out of your life and check it out.  I guarantee regardless of your political inclination you will get something out of it. It will give you a reason to really think about what sort of society we are becoming.

So sometime yesterday I noticed a flood of error messages hitting my email.  They are messages that mean that for some reason my local server was unable to pull mail from my host, IX Webhosting.  This happens from time to time.  Sometimes the issue is local, sometimes the issue is at IX but typically it doesn’t last very long so I just ignored it.

Well when I checked back on my mail later I saw I had 300+ of these error emails so I figured it was time to see what was up.  Evidently my good pals at IX, whom I have had various issues/rants about over the last 5 years had a major hardware problem.

The way it was described was they had “multiple” drives fail simultaneously in their SAN (storage area network)  This SAN evidently was shared storage for 150 or more servers at IX, meaning every single one of these servers was now dead in the water.

In IT it is quite common for hard drives to be built in such a way thet they offer redundancy since they are the most failure prone device in a system.  Tiny platters spinning at 5000-15000 rpm’s 24 hours a day, 7 days a week are going to die sooner or later.

To combat this, RAID was invented (redundant array individual disks).  In layman terms, you tie several hard drives together in such a way that if one fails your data is intact, you simply replace the failed drive and all is fine again.  The most common level of RAID is RAID level 5 which is what I just described.

However in a huge ISP RAID 5 is simply not enough protection since data storage is so critical.  There are higher levels of RAID.  Level 6 allows for up to two drives to die at the same time, something which is statistically minute however an ISP needs that level of protection just because the hard drive arrays are depended on by literally thousands of customers.

Well in IX’s explanation of the failure they used the failure as “multiple” disks.  I would like to know what multiple means. Because if multiple means two drives failed at the same time, which as I said is very, very, rare, that means they had their SAN only using RAID 5, which is very irresponsible and poorly designed.  Of course they would never admit that publically.

So anyway, every single thing in my personal internet web sphere has been offline for 24 hours and counting.

The IX status page has been DESTROYED by customers bitching non-stop about the outage as you can imagine.  I saw no need to pile on, since being in IT I can sympathize with the engineers that are trying to get this data disaster monster back in it’s cage.  It doesn’t make me any happier though about IX overall as a web hosting company.

Time and time again they have let me and many others down.  The only thing that has kept me there is the massive task I would face relocating my earth mover sized pile of content to another web hosting provider.  Depending on how much longer this drags on, the effort/reward formula to move my sites might just tip the other way.

I got my new Gunnar computer glasses on Friday.  I wore them during the weekend while I was on the computer and brought them into work today.  So far I like them.  They are so light I hardly know I am wearing them.  Visually, they brighten, focus and slightly magnify the images on the screen.

Things just look better through the Gunnars.  I plan to wear them full time at work.  I won’t be able to give a full report card until I give my eyes time to adjust.  I am hoping that my distance detail vision slowly will come back around.