Blog | Admin | Archives | Heatmap | Recent | Random | Plan
    • Instagram Image
    • Red Lentil Chorizo Soup
    • Val Thorens majesty.
    • Just the Rosetta Stone.

Upgrade to Ubuntu 16.04 LTS

A few days ago, I was poking about “micro”, the AWS EC2 server behind this site. I noticed that a new LTS release of Ubuntu was out so I decided to upgrade.

Unfortunately, I got distracted amid the upgrade and forgot about it, and then it took my brother to inform me that things had gone awry: every page load was returning “502/Bad Gateway”. So, just like last time, I had to dig in to figure out what was going wrong. I started looking into it and the problem ended up being multi-faceted.

First, I needed to finish off the install which I had rudely interrupted by rebooting the server while the do-release-upgrade was stuck at a prompt. Whoops! Fortunately, apt-get is nice enough these days to tell you the invocation you need to resume the upgrade — something like dpkg -a. Regardless, I got that resumed and finished up, then I set about seeing if everything was working.

Well, of course it wasn’t! First of all, nginx was set to work with php5-fpm, but this new release ships with php7, which has been put under the more generic name php (which seems like a good move, even if it’s backwards incompatible, because it allows for compatibility going forward, whereas the old method did not).

The first step was to update the nginx configs so that it would talk to the right unix domain socket to communicate with the upgraded php-fpm package. Once that was done, however, the pages started showing up blank, but with HTTP/200 responses, as if everything was working fine. Some searching led me to the regular place where all sysadmin questions go to be answered: serverfault.

Sure enough, this was exactly the issue I had, and adding that line to my nginx config made the websites start loading again.

Now, I just have to figure out the email situation again…

Dovecot + Maildir + Ubuntu 14.04 LTS Upgrade

I recently upgraded the server behind this site to Ubuntu 14.04 LTS from 12.04 LTS (only about a year late!)

A few things went awry (the PHP install couldn’t talk to MySQL, for example), but a reboot cleared that right now. However, one piece remained broken. Mail wasn’t properly being delivered.

I’m don’t use this server for mail much myself, but some people (like my brother) do. It was broken, and I had no idea why. It didn’t help that the last time I touched the config was over a year ago. It also doesn’t help that mail server setup is basically a dark art. Regardless, after a few days of poking at it and “hoping the problem would go away”, I decided to go at it again today. Hopefully by writing this down I’ll remember a bit more about my setup, but if not, at least I’ll have this handy reference when I forget it again.

The way mail is set up on this server is that Postfix listens for incoming SMTP traffic, which it then forwards to Dovecot for delivery. Dovecot is set up to use the Maildir format, but instead of storing the maildirs in users’ home directories, it stores them in /var/mail/ since not all the users even have home directories. It made sense at the time, and I think it still makes sense now!

At any rate, with Dovecot’s upgrade came a problem, and after digging around, I saw in the logs that Dovecot was unable to deliver mail to /home//Maildir. Of course, those directories didn’t exist, so of course it was failing! However, I had set up the mail_location to be /var/mail/, so what was going on?

It turns out that Dovecot’s Ubuntu distro added a new configuration file, /etc/dovecot/conf.d/99-mail-stack-delivery.conf, which set the mail_location back to /var/mail/, and apparently the last place to set it wins (it was previously set in /etc/dovecot/conf.d/10-mail.conf). I commented out the offending line, and by tailing /var/log/mail.log, I could see that mail was once again delivering.

Now, I don’t know if this means it’s fixed; my brother will have to let me know. Regardless, it’s at least less broken now.

Upgrading from Ubuntu 9.04

Among other things, I’ve let minimus go far too long without upgrades. It’s still running Ubuntu 9.04, which, while working, is old enough to be unsupported, especially since it’s not a long-term-service (LTS) release.

Upgrading from an unsupported release is, unfortunately, not officially supported, but some intrepid souls have figured out how to make it work anyway. Using answers from this thread on AskUbuntu, I have been able to get the process started. Of course, it remains to be determined how this will finish.

Sharks or glory lay ahead.

xkcd: Success

A Glorious Hack

Today, a buddy mentioned that his blog wasn’t loading. His blog was hosted on one of the servers behind silverfir.net, minimus. It’s the one with the really interesting setup: Windows Server running Ubuntu Linux in a VM.

I luckily remembers how to log into the Windows server via remote desktop and poking around I discovered that something had gone wonky with the VM image. A reboot + disk scan didn’t fix the issue, so I created a new VM image and mounted the same disk images and things seems to start working reasonably well again.

Except that the port forwarding wasn’t working. Minimus is NAT’ed behind a very high-speed residential line, and the dd-wrt router it’s behind forwards a few relevant ports to the right place: 22 and 80 come to mind. Somehow, perhaps because of a new (virtual) MAC address, the forwarding wasn’t working any more.

I wanted to get the sites back online, but I didn’t have access to the router’s admin interface, so I came up with a glorious hack. Here’s what I did:

(1) ssh with port forwarding from minimus to nexus to forward port 5050 on nexus to port 80 on minimus: ssh -R 5050:localhost:80 nexus.silverfir.net
(2) on nexus, create an nginx config that forwards all the relevant sites to port 5050
(3) point the dns entries for all the relevant sites to nexus instead of minimus

It is working surprisingly well, but of course this is a super fragile state where one connection dropping will mean all the sites become unavailable again. If you care much about content hosted on minimus, you should probably take this chance to back it up.

Ten Years of Silverfir.net

More than ten years ago, I registered the silverfir.net domain name. From its humble beginnings as debian server at my parents house, to several stints at various friends’ houses, to the multi-state distributed network it is now (WA, CA, VA), it has served me and my friends well — and will continue to, long into the future.

In this time, silverfir has been the home of my blog and many friends’ blogs and personal website projects. Its various servers have served as file servers, a development platforms, web hosts, and learning devices. It has been hacked once, but there wasn’t much interesting to see, I’m glad to say.

From the whois records:

Domain Name: SILVERFIR.NET
Created On: 20-Mar-2003 00:01:37 UTC
Last Updated On: 25-Dec-2011 01:27:54 UTC
Expiration Date: 19-Mar-2015 23:01:37 UTC
Sponsoring Registrar: Vitalwerks Internet Solutions, LLC / No-IP.com
Registrant Name: McElroy, Ryan

Blog Optimization

In the last two days, I

  1. Changed my blog’s MySQL tables storage engines from the MyISAM to InnoDB
  2. Installed the WordPress Memcache Plugin to mimimize database queries (16-25 queries reduced to 2-7)
  3. Installed APC (Alternative PHP Cache) to reduce PHP bytecode compilation overhead. As a result, all PHP sites on mimimus should be faster.

In addition, I did some general cleaning up and upgrading of software on minimus and nexus.

Altogether, these changes reduce the typical Checksum Arcanius page load from 2.5-3.5 seconds to 0.5-1.5 seconds, a 2-7x improvement.

These are very easy steps to take — I would suggest them to anyone running WordPress. Step-by-step directions follow (assuming Ubuntu Linux):

  1. For each table in your blog’s database, execute the following SQL via a mysql client instance, phpMyAdmin, etc:
    ALTER TABLE <tablename> ENGINE = InnoDB;
  2. Install memcache:
    sudo apt-get install memcache
  3. Download the WordPress Memcache Plugin and place it in your wp-content directory. That is all you have to do to get memcache support in WordPress!
  4. Install APC:
    sudo apt-get install php-apc
  5. Restart Apache:
    sudo /etc/init.d/apache2 restart

Very simple steps with a very high payoff.

The Trend

The Trend is Clear

As I have become more involved with Facebook and Twitter, the number of idea that actually mature enough for me to blog about them go down. Also, I’m not taking enough time to review my thoughts and write about them.

My first resolution of 2010: I resolve to do blog more. I need that time to reflect reflect on my life. Status updates are not sufficient.