RILOE to the Rescue
After attempted to run the open-source Math software Sage on Frankenputen and being harshly greeted by an error message claiming that libc was out of date, Bobby and I decided that it was time to upgrade the Dapper Drake edition of Ubuntu Linux that was running the server to the modern Gutsy Gibbon edition.
The first attempt was an abject failure. Used to Debian’s nearly perfect reliability, I thought I could get away with upgrading direct from Dapper to Gutsy by simply changing my sources.list file, the running apt-get update followed by apt-get dist-upgrade. Well, this series of actions ended quite poorly — what was left over was a pretty broken, not very-installed hybrid of Dapper and Gutsy. So, after some tinkering and with some help from Bobby, we managed to revert — eventually — to Dapper. Bobby then set about doing a more incremental upgrade, stepping from Dapper to Edgy to Feisty and then to Gutsy (notice the progression of letters). This apparently worked, almost.
Well, it did work, but at the same time something else broke. Namely, the kernel version shipped with Gutsy — 2.6.22 — and a package called evms do not coexist peacefully. The result was a system that would being the boot procedure and then get stuck in an infinite loop of error messages. The key services — such as SSH — that we would normally use to try to fix the problem never came up, so all hope was lost, right? Well, not quite.
Back when Dan purchased all the smograsbord (sp?) of technology that now comprises Frankenputen, the computer currently responsible for serving much of silverfir.net’s content, he had the foresight to purchase an nice little piece of technology called a RILOE card. RILOE stands for Remote Insight: Lights Out Edition. Basically, its a video card with a network port on the back. It runs a web server, which allows administrators to access a variety of functions on the server as if the administrator where at the computer physically. Even more spectacularly, it has a virtual local console — basically a virtual screen that shows exactly what would be on the real screen. So when the server was dead to the rest of the world, after a little tinkering, I was able to log in and watch the system boot from the ground up — even the BIOS messages are visible! Its really like being at the computer in pretty much every pertinent way expect physical proximity.
Watching the system come up gave some error messages which Google turned into problem-solving tips. Then, by booting an older kernel, uninstalling evms, and rebooting, the machine was back to working as good as old. But with a bunch of upgrades of course!