WordPress and CPU utilization problems on shared web hosting

Regular blog readers I apologize, this post is targeted at getting picked up by others looking for info regarding how to tame WordPress CPU use in a shared hosting environment.

A month ago I arranged to migrate/host a website for a worldwide calisthenics/bar exercise group that I participate in.  The site needed a refresh and I thought WordPress would be a good back end to host the site since it is so flexible yet powerful.  After about a month of getting the new site ready we opened the doors in early February.

Well within several hours of launch the site was shut down by my shared web hosting provider due to very high CPU utilization.  They said I was averaging 170 cpu minute cycles, the limit for shared hosting for them is 60.  They identified a couple plug ins that were causing a lot of cpu usage, WP-SPAMFREE and Nextgen Photo Gallery.    I told them I would remove the plug ins.

Well this began more than a month long back and forth battle between my web host and I.  They would shut down the domain, I would get it turned back on,  do more tweaks and then it would get shut down again.  Finally, I have done enough tweaking to get the site safely below the CPU threshold.

When I first launched the site in addition to the high CPU use I had a lot of bandwidth and hits as well.  Over 6 gig of traffic was recorded the first day.  The amount of unique visitors wasn’t crazy, around 1500 but the amount of overall hits reported for those visitors seemed disproportionately high, over 200,000.

My first work was done streamlining the plug ins as I mentioned, eliminating the use of any that were CPU hogs which helped a bit but not nearly enough.

Next I scoured the internet for suggestions on making WP play nice with shared hosting.  One of most frequent suggestions was to install a caching plug in.  A cache plug in saves resources by presenting most visitors with a cached, flat file instead of a dynamic page that has to be created with database calls each and every time.

I first tried WP Super Cache but later switched to W3 Total Cache as it is well established as the most powerful caching engine out there.  Enabling the caching plug in made a major dent in the amount of bandwidth and cpu that I was utilizing but not nearly enough, I was still 15-20 cpu minutes over the threshold.

One of the reasons W3 is awesome is it has direct support for other traffic optimizing services, like CDN’s. (content delivery networks)  A CDN is a service whose sole purpose is to house the larger files that make up your website and serve them up very quickly, offloading the traffic from your shared host to them seamlessly.  I utilized Amazon Cloudfront as my CDN.  There is a small cost associated with the service but nothing painful.

Setting up the CDN made a huge difference in the bandwidth used by the site which was averaging somewhere in the neighborhood of 2 gigs a day.  After implementing the CDN the traffic dropped to 200 MEGS, wow!  I thought I finally had the problem licked but alas, once again my host shut down the site, saying my utilization was much better but still above the red line.

I implemented some more changes as I was still seeing an unusually high amount of hits for the amount of unique visitors the site was getting.  I set up a robots.txt file to try to stop the site from being crawled incessantly by web search spiders.  I turned on the WP setting that discourages search engines as well as following a few other tweaks I obtained from others posting on the problem.   Again, they helped, but not enough.

After a month of this crap I was about ready to give up, resigning myself to having to move the site to more expensive hosting like wpengine.com. (great place to host WP)  However before raising the white flag I decided to implement one more idea I read about, CloudFlare.

CloudFlare is a service designed to analyze and scrub your website traffic before it even hits your web host.  W3 again directly supports the service.  The way it works is you actually need to change your DNS for the domain to point to the CloudFlare service.  Yes, it sounded problematic to me as well, but it has been seamless thus far.

So when traffic for your site comes in CloudFlare sees it, determines if it is legitimate or bots, crawlers or other undesirables.  If it is legit it passes the traffic along, if it isn’t it kills it on the spot.

Well this has been the magic bullet.  Since implementing CloudFlare my cpu usage has plummeted.  Cloudflare has been eliminating over 60% of the traffic coming in, it was all non-human requests that were hammering the site.  My most recent stats showed that 600 visitors generated, 3500 hits which makes much more sense.  It is amazing just how much of an impact bots were having on the site.

The best part of all is the cost, zero.  I am able to use the free level of service.  CloudFlare has higher levels of service that would be useful for bigger operations but for me, free works just fine.

Here are some useful plug ins that helped me streamline the site:

P3 (Plug in Performance Analyzer) – this loads various pages in your site and creates a clear report of which plug ins on your site are killing performance.  It also lets you know how many SQL calls are required to load an average page.  Obviously the less calls made, the better.

Plugin Organizer – this allows you to selectively disable plug ins for only certain areas on your site.  In my case it allowed me to turn off certain stuff when running the SimplePress forum on my site. But turns them on otherwise.

WP-Optimize – Let’s you see details of your WP database and clean out unnecessary tables left by old plug ins, post revisions and more.  This allowed me to greatly reduce the size of my WP db.

So to recap in a Cliff Notes way, if you are having CPU warnings from your host I would advise you to do the following:

Install W3 total cache.  There are various resources on the web that can guide you on the recommended settings.

Utilize a CDN.  It will speed up your site and reduce usage on your shared hosting

Set up a robots.txt file and discourage search engines in WP

Set up CloudFlare!  This service is simply awesome.  It is like putting a condom on your website.

These changes knocked my CPU usage down 70% from where I started and saved me from having to spend a bunch more of money on my hosting.

Of course my web host said the only way I could fix the problem was by migrating to their dramatically more expensive dedicated hosting.  I guess they were wrong.