Checksum Arcanius

I finally bit the bullet and used lets encrypt to get certificates for several of the domains hosted by this server.

It was relatively straightforward in the end — though the program didn’t know how to completely automate the setup I had, the instructions were pretty good and the program was pretty clear in what it was doing.

I consulted a post on the nginx blog for guidance as well.

Say hello to https://arcanius.silverfir.net!

I’ve just finished upgrading the server behind this site to the latest Ubuntu LTS release, 20.04 “Focal Fossa”.

I think I’ve ironed out all the issues, but if you see anything weird, give me a shout in the comments or via other means.

I’ve been poking around the server hosting this site for the last few days and just took the time to upgrade to the latest LTS release of Ubuntu (18.04). Things went smoothly, I think — the only manual intervention required was moving php-fpm from 7.0 to 7.2, which involved only a straightforward copy of the old config files.

Just before the upgrade, I moved to a model of a separate php-fpm instance for each of the sites hosted on this server. Each one is running as it’s own user, so in theory the security guarantees should be much stronger (at most, a single site might be compromised — there should be no way to damage a second site if a first site has a vulnerability).

As always, if you notice anything amiss, please let me know.

A few days ago, I was poking about “micro”, the AWS EC2 server behind this site. I noticed that a new LTS release of Ubuntu was out so I decided to upgrade.

Unfortunately, I got distracted amid the upgrade and forgot about it, and then it took my brother to inform me that things had gone awry: every page load was returning “502/Bad Gateway”. So, just like last time, I had to dig in to figure out what was going wrong. I started looking into it and the problem ended up being multi-faceted.

First, I needed to finish off the install which I had rudely interrupted by rebooting the server while the do-release-upgrade was stuck at a prompt. Whoops! Fortunately, apt-get is nice enough these days to tell you the invocation you need to resume the upgrade — something like dpkg -a. Regardless, I got that resumed and finished up, then I set about seeing if everything was working.

Well, of course it wasn’t! First of all, nginx was set to work with php5-fpm, but this new release ships with php7, which has been put under the more generic name php (which seems like a good move, even if it’s backwards incompatible, because it allows for compatibility going forward, whereas the old method did not).

The first step was to update the nginx configs so that it would talk to the right unix domain socket to communicate with the upgraded php-fpm package. Once that was done, however, the pages started showing up blank, but with HTTP/200 responses, as if everything was working fine. Some searching led me to the regular place where all sysadmin questions go to be answered: serverfault.

Sure enough, this was exactly the issue I had, and adding that line to my nginx config made the websites start loading again.

Now, I just have to figure out the email situation again…

A couple years back, I moved from a full tower for my file server to a small HP Proliant MicroServer (I’ve been very happy with this move). When I made that move, I went from 4x750GB in a RAID 10 configuration to only 2x750GB in a RAID 1 configuration. However, I keep taking photos and I noticed I was running out of space. Since I had the extra Hard Drives just sitting around, I decided to put them back into use. the hardest part was that there are only four HDD slots in the server, and I was using one of them for the boot drive.

However, I had long noticed that there was a 5.25 inch drive bay in the server that was conspicuously unused. All I needed was an adapter. It took me two years to finally get around to this project, but today was the day! I found a drive bay adapater and SATA power and data cables, then got to work. From a physical perspective, it was fairly straightforward once I had all the parts; the hardest part ended up being finding the extra SATA port on the motherboard.

I was a bit wary when I booted the thing — the hard drives I used had previously been a part of the 1.5TB array that the other two came from. Fortunately, everything went smoothly. It was then a rather simple matter of finding this excellent guide on ServerFault to rebuild the array into RAID 10 once again.

Yay technology; internet FTW!

Based on an answer on serverfault, but modified to work on linux:

Use this one-liner to get the bash and sshd processes currently hanging onto the old screen.

ps aux | grep $(ps aux | grep [s]creen | grep -o 'pts[^ ]*') | grep 'sshd\|bash'

Kill these processes and screen is recoverable before the long and arbitrary timeout.

I recently upgraded the server behind this site to Ubuntu 14.04 LTS from 12.04 LTS (only about a year late!)

A few things went awry (the PHP install couldn’t talk to MySQL, for example), but a reboot cleared that right now. However, one piece remained broken. Mail wasn’t properly being delivered.

I’m don’t use this server for mail much myself, but some people (like my brother) do. It was broken, and I had no idea why. It didn’t help that the last time I touched the config was over a year ago. It also doesn’t help that mail server setup is basically a dark art. Regardless, after a few days of poking at it and “hoping the problem would go away”, I decided to go at it again today. Hopefully by writing this down I’ll remember a bit more about my setup, but if not, at least I’ll have this handy reference when I forget it again.

The way mail is set up on this server is that Postfix listens for incoming SMTP traffic, which it then forwards to Dovecot for delivery. Dovecot is set up to use the Maildir format, but instead of storing the maildirs in users’ home directories, it stores them in /var/mail/ since not all the users even have home directories. It made sense at the time, and I think it still makes sense now!

At any rate, with Dovecot’s upgrade came a problem, and after digging around, I saw in the logs that Dovecot was unable to deliver mail to /home//Maildir. Of course, those directories didn’t exist, so of course it was failing! However, I had set up the mail_location to be /var/mail/, so what was going on?

It turns out that Dovecot’s Ubuntu distro added a new configuration file, /etc/dovecot/conf.d/99-mail-stack-delivery.conf, which set the mail_location back to /var/mail/, and apparently the last place to set it wins (it was previously set in /etc/dovecot/conf.d/10-mail.conf). I commented out the offending line, and by tailing /var/log/mail.log, I could see that mail was once again delivering.

Now, I don’t know if this means it’s fixed; my brother will have to let me know. Regardless, it’s at least less broken now.