Archives March 2014

Here I am going on my third day of internet darkness.   In a way I feel somewhat lucky that none of the sites I run/maintain are used for a real business.  When I read the comments from IX customers that are literally losing fistfuls of money and customers from this outage I feel badly for them.

This outage reenforces my dislike of MASSIVE consolidation of server resources, at least for things I manage personally.  Sure it’s really cool to have 10 servers shrunk down into one server, however if something just happens to go haywire with that one server, 10 servers go down with it.  That is why for mission critical things in our environment I use a traditional server model with individual RAID protected storage.  Call me old fashioned but I just don’t feel great having too many eggs in one basket.  I like my castrophic failures to be contained in nice little pockets of chaos instead of network wide.

The status blog for IX Webhosting is lit up with over 1500 comments from customers that are besides themselves about the slow recovery from this failure.  I find myself chuckling at the pleas to DO MY SERVER FIRST!  This is probably the longest outage I have had of my internet presence since it was implemented back in the 90’s.   Yes it isn’t good to be cut off from email for this long since it is my most heavily used and preferred vehicle of communication, but there is literally nothing I can do about it at this point.

Well this is an unusual occurrence, doing a blog posting in Wordpad.  I will be outlining the reason for this later in the entry.

I got home Friday night and noticed the pool cage rescreening job had been completed.  My eyes were immediately drawn to the two corners of the cage which had small wrinkles in the screen.  Other than that the rest of the screen looked really good.

We had a beautiful morning for the race I timed Saturday morning.  The turnout for the event was very small with somewhere around 175 people signing up.  This race takes place on the same weekend as a large half marathon in Fort Myers so it is set up for failure.  It really needs to be moved to some other weekend in the future.

The race was so small that I was able to shoot some aerial video with the Phantom as finishers streamed across the finsih line.  Both spectators and participants were fascinated by the buzzing, 4 bladed camera platform above.  The video I captured turned out pretty well.  I hope to be able to do this more often in the future.

On Saturday afternoon after finishing up post race work I went to pick up the dogs.  I told Ali I could keep them overnight.  It worked out well since she was one of those people running the Fort Myers race on Sunday.

Cindy and I decided to spend Saturday night at the house instead of hitting the movies.  I had a fresh Netflix rental to watch anyway, Rush.  Rush has the guy that plays Thor in it but if you ask me, the story, at least the more compelling part of the story is about Nikki Lauda, Thor’s nemesis in Formula 1 racing.  I remember hearing Nikki’s name growing up during ABC Wide World of Sport broadcasts but I had no idea he went through the hell depicted in the film.

I found the movie to be good, quite surprising since I could care less about racing in general.  I’d give it a strong B+

Sunday morning started off slow, it was very damp and foggy.  We scrapped another biking/swimming brick for another run around the track instead.  I was surprised that simply moving from lane 1 to lane 2 on the track added nearly a 1/4 mile to the total distance according to my GPS.  My Garmin said we covered 3.37 miles at roughly an 8:30 pace doing 12 laps around the oval.

The girls had fun hanging out at the house as usual.  They just love being able to walk around outside without any major obstacles.

The pool screen guy stopped by Sunday afternoon to collect the balance of the payment.  I pointed out the screen wrinkles.  He said he was hoping a day or two in the sun would have smooth them out. (really?)  He said he would be back on Monday to take care of them.

While he was there I asked him  about the technique used in rescreening an entire cage.  Since I have spent many irritating hours replacing screen panels I was curious to hear how you do it on a large scale.  He said they attach one end and then unroll the screen to cover an entire row of panels, they pull it tight, drop in the spline and then cut the screen.  It saves a whole bunch of cutting and measuring.

On Sunday evening Cindy and I watched Inequality for All, a film produced by Robert Reich, a name/face you may recognize.  Robert’s most famous position was as labor secretary under Bill Clinton. In recent years he has been popping up all over the place making commentary on the growing wealth disparity in the United States, both why it has been accelerating the last 30 years and why it is a huge problem.

The documentary, which is available for streaming free on Netflix was 90 minutes of some stuff I already knew and a lot of stuff I didn’t.  The start of this canyon size gap of wealth disparity started pretty clearly in the late 1970’s when the wages for the majority of the population stopped following the same general upward curve as the economy.  Wages flat lined while the economy continued to generate more and more revenue.  The overwhelming percentage of that additional revenue went directly into the pockets of an amazingly small amount of individuals.

The film will do a much better job of explaining the issue than I will regurgitating the facts here but let me simply say that any logical human being should see why this trend is unsustainable and self destructive in the long term, unless the goal of our country is to exist in a feudal society.

Once the ultra-rich cross a certain point of control in politics, reversing that trend becomes a task of monumental portions.  How do you change policies that benefit massive corporations if the massive corporations have bought enough influence in Washington to ensure nothing gets done?

Despite Roger’s small stature, I don’t think he is even 5 feet tall, he has a big voice on this subject.  I have followed him on Facebook for quite awhile and generally find myself shaking my head in agreement to most of what he says.

He does have some ideas that seem a bit far out there, even for me, like wanting to raise the minimum wage nearly 50% beyond the new levels Obama had mentioned ($10 and hour).  Robert wants $15 an hour as the new minimum hourly wage.

Robert says without a strong middle class, the country will rot from within, as it is the middle class that does the majority of the spending that drives consumption of products and services in our country.  His thinking is if people make a higher minimum wage, more of that spending can occur.  Of course there are counterpoints to this theory such as increasing the minimum wage could cause even greater unemployment or drive up inflation.

I did find it quite interesting that the one woman featured in the documentary was a Costco employee,  a big chain that sells goods warehouse style like Wal-mart does with Sam’s Club.  The retail market for the most part has been operating under the guise that there is absolutely no way that a retailer can offer employees decent wages and still be profitable.  Well Costco is a huge middle finger to that concept.  The woman was making $21+ an hour.  It can be done.

Please take an hour and a half out of your life and check it out.  I guarantee regardless of your political inclination you will get something out of it. It will give you a reason to really think about what sort of society we are becoming.

So sometime yesterday I noticed a flood of error messages hitting my email.  They are messages that mean that for some reason my local server was unable to pull mail from my host, IX Webhosting.  This happens from time to time.  Sometimes the issue is local, sometimes the issue is at IX but typically it doesn’t last very long so I just ignored it.

Well when I checked back on my mail later I saw I had 300+ of these error emails so I figured it was time to see what was up.  Evidently my good pals at IX, whom I have had various issues/rants about over the last 5 years had a major hardware problem.

The way it was described was they had “multiple” drives fail simultaneously in their SAN (storage area network)  This SAN evidently was shared storage for 150 or more servers at IX, meaning every single one of these servers was now dead in the water.

In IT it is quite common for hard drives to be built in such a way thet they offer redundancy since they are the most failure prone device in a system.  Tiny platters spinning at 5000-15000 rpm’s 24 hours a day, 7 days a week are going to die sooner or later.

To combat this, RAID was invented (redundant array individual disks).  In layman terms, you tie several hard drives together in such a way that if one fails your data is intact, you simply replace the failed drive and all is fine again.  The most common level of RAID is RAID level 5 which is what I just described.

However in a huge ISP RAID 5 is simply not enough protection since data storage is so critical.  There are higher levels of RAID.  Level 6 allows for up to two drives to die at the same time, something which is statistically minute however an ISP needs that level of protection just because the hard drive arrays are depended on by literally thousands of customers.

Well in IX’s explanation of the failure they used the failure as “multiple” disks.  I would like to know what multiple means. Because if multiple means two drives failed at the same time, which as I said is very, very, rare, that means they had their SAN only using RAID 5, which is very irresponsible and poorly designed.  Of course they would never admit that publically.

So anyway, every single thing in my personal internet web sphere has been offline for 24 hours and counting.

The IX status page has been DESTROYED by customers bitching non-stop about the outage as you can imagine.  I saw no need to pile on, since being in IT I can sympathize with the engineers that are trying to get this data disaster monster back in it’s cage.  It doesn’t make me any happier though about IX overall as a web hosting company.

Time and time again they have let me and many others down.  The only thing that has kept me there is the massive task I would face relocating my earth mover sized pile of content to another web hosting provider.  Depending on how much longer this drags on, the effort/reward formula to move my sites might just tip the other way.

I got my new Gunnar computer glasses on Friday.  I wore them during the weekend while I was on the computer and brought them into work today.  So far I like them.  They are so light I hardly know I am wearing them.  Visually, they brighten, focus and slightly magnify the images on the screen.

Things just look better through the Gunnars.  I plan to wear them full time at work.  I won’t be able to give a full report card until I give my eyes time to adjust.  I am hoping that my distance detail vision slowly will come back around.