Checksum Arcanius

Today, a buddy mentioned that his blog wasn’t loading. His blog was hosted on one of the servers behind silverfir.net, minimus. It’s the one with the really interesting setup: Windows Server running Ubuntu Linux in a VM.

I luckily remembers how to log into the Windows server via remote desktop and poking around I discovered that something had gone wonky with the VM image. A reboot + disk scan didn’t fix the issue, so I created a new VM image and mounted the same disk images and things seems to start working reasonably well again.

Except that the port forwarding wasn’t working. Minimus is NAT’ed behind a very high-speed residential line, and the dd-wrt router it’s behind forwards a few relevant ports to the right place: 22 and 80 come to mind. Somehow, perhaps because of a new (virtual) MAC address, the forwarding wasn’t working any more.

I wanted to get the sites back online, but I didn’t have access to the router’s admin interface, so I came up with a glorious hack. Here’s what I did:

(1) ssh with port forwarding from minimus to nexus to forward port 5050 on nexus to port 80 on minimus: ssh -R 5050:localhost:80 nexus.silverfir.net
(2) on nexus, create an nginx config that forwards all the relevant sites to port 5050
(3) point the dns entries for all the relevant sites to nexus instead of minimus

It is working surprisingly well, but of course this is a super fragile state where one connection dropping will mean all the sites become unavailable again. If you care much about content hosted on minimus, you should probably take this chance to back it up.

More than ten years ago, I registered the silverfir.net domain name. From its humble beginnings as debian server at my parents house, to several stints at various friends’ houses, to the multi-state distributed network it is now (WA, CA, VA), it has served me and my friends well — and will continue to, long into the future.

In this time, silverfir has been the home of my blog and many friends’ blogs and personal website projects. Its various servers have served as file servers, a development platforms, web hosts, and learning devices. It has been hacked once, but there wasn’t much interesting to see, I’m glad to say.

From the whois records:

Domain Name: SILVERFIR.NET
Created On: 20-Mar-2003 00:01:37 UTC
Last Updated On: 25-Dec-2011 01:27:54 UTC
Expiration Date: 19-Mar-2015 23:01:37 UTC
Sponsoring Registrar: Vitalwerks Internet Solutions, LLC / No-IP.com
Registrant Name: McElroy, Ryan

In the last two days, I

1. Changed my blog’s MySQL tables storage engines from the MyISAM to InnoDB
2. Installed the WordPress Memcache Plugin to mimimize database queries (16-25 queries reduced to 2-7)
3. Installed APC (Alternative PHP Cache) to reduce PHP bytecode compilation overhead. As a result, all PHP sites on mimimus should be faster.

In addition, I did some general cleaning up and upgrading of software on minimus and nexus.

Altogether, these changes reduce the typical Checksum Arcanius page load from 2.5-3.5 seconds to 0.5-1.5 seconds, a 2-7x improvement.

These are very easy steps to take — I would suggest them to anyone running WordPress. Step-by-step directions follow (assuming Ubuntu Linux):

1. For each table in your blog’s database, execute the following SQL via a mysql client instance, phpMyAdmin, etc:
   ALTER TABLE <tablename> ENGINE = InnoDB;
2. Install memcache:
   sudo apt-get install memcache
3. Download the WordPress Memcache Plugin and place it in your wp-content directory. That is all you have to do to get memcache support in WordPress!
4. Install APC:
   sudo apt-get install php-apc
5. Restart Apache:
   sudo /etc/init.d/apache2 restart

Very simple steps with a very high payoff.

The Trend is Clear

As I have become more involved with Facebook and Twitter, the number of idea that actually mature enough for me to blog about them go down. Also, I’m not taking enough time to review my thoughts and write about them.

My first resolution of 2010: I resolve to do blog more. I need that time to reflect reflect on my life. Status updates are not sufficient.

Silverfir.net is up for some big changes in the near future. Dan has ordered up a new system, and I brought a couple of 750 gb hard drives from nexus back from California  (although one of them is throwing errors and probably won’t be used). In the end, the yet-to-be-named server will take over the primary web serving duties for silverfir.net.

The switchover process will start tomorrow, when silverfir.net’s web service will go down while the current server, Frankenputen, is moved to Dan’s place, where it will be easier to sync with the new server. During this move, barring some sort of amazing in-car power and internet source, silverfir.net will be inaccessible. Most likely, it will be down for a few daytime hours tomorrow, so don’t be too shocked if you can’t access it for a while.

My previous plans of retiring Frankenputen may have to be accelerated. Today, Frankenputen stopped serving for an unknown reason. I was able to resuscitate it by forcing a restart via the excellent Remote Insight console, but I’m beginning to wonder how much longer the beast will last.

I’ve started twittering. For a long time, I had no interest in the phenomenon, but recently I’ve noticed that I’m not blogging as much as I want to, while I still have lots of ideas I want to write about. In a perfect world, a lot of these ideas would become full blog posts; however, until I rededicate myself to making that happen, my hope is that at least the ideas won’t die unheard — they can now live on as tweets. As Mark Cuban said, “Tweets are the blog posts you thought about writing, but didn’t feel they had enough substance.” (Hat tip Theo)

So, if you’re into that sort of thing, you can follow me on Twitter, or just occasionally visit my blog to see my most recent tweets.

I’m also considering a “daily digest” of my tweets as actual blog posts; there is a plug-in to do that.