Update 2008/08/22: There’s actually a simpler command to create the database, once MySQL is secured and the account exists:

mysqladmin create newdatabase -u existinguser -p.

To test Movable Type, I needed a new MySQL installation on a CentOS 5.2 (equivalent to Red Hat Enterprise Linux 5.2) system. Here’s a crib sheet with the steps I took to set up a new MySQL installation.

Get and Start the Software

• yum install perl-DBD-MySQL mysql-server # Install MySQL server and the DBD perl module that Movable Type needs to talk to it.
• service mysqld start # Start MySQL server.
• chkconfig mysqld on # Set mysqld (the MySQL ‘daemon’, or server) to run at boot in future.

Secure MySQL

MySQL uses internal accounts which are totally separate from UNIX accounts. My MySQL installation came with 3 distinct root accounts (without passwords); a RHEL4 system configured MySQL with a pair of anonymous accounts! In the past, I have used mysqladmin, which is quicker, but apparently it doesn’t do a complete job. MySQL offers [instructions on how to secure the initial accounts](http://dev.mysql.com/doc/refman/5To test Movable Type, I needed a new MySQL installation on a CentOS 5.2 (RHEL 5) system. Here’s a crib sheet with the steps I took to set up a new MySQL installation.

Get and Start the Software

• yum install perl-DBD-MySQL mysql-server # Install MySQL server and the DBD perl module that Movable Type needs to talk to it.
• service mysqld start # Start mysqld (the MySQL ‘daemon’, or server).
• chkconfig mysqld on # Set mysqld to run at boot in future.

Secure MySQL

MySQL uses internal accounts which are totally separate from UNIX accounts. My MySQL installation came with 3 distinct root accounts (without passwords); a RHEL4 system configured MySQL with a pair of anonymous accounts! The MySQL RPM suggests securing the default accounts with mysqladmin, but the website points out that mysqladmin doesn’t get all the accounts. Fortunately MySQL offers instructions on how to secure the initial accounts manually.

mysql> select host, user from mysql.user;
+----------------+------+
| host           | user |
+----------------+------+
| 127.0.0.1      | root | 
| localhost      | root | 
| mmm.reppep.com | root | 
+----------------+------+
3 rows in set (0.00 sec)

In WordPress, each blog has its own account and database (that’s how I configure them, anyway). In Movable Type, a single account & database will be used for my whole Movable Type installation, which makes administration simpler.

• Secure both root accounts by setting strong passwords.
• Delete both anonymous accounts.
• Create a new account for the blog.

To make sure I really did configure a required password for root, I logged out of MySQL and then tried to login without a password (which is how got in initially). This failed, telling me I had successfully disabled passwordless root access. Then I logged in as root with a password, to continue setting up MySQL tables. Note that I never supply passwords on the command line, because that’s insecure. Instead I supply the password when prompted by the mysql command, which keeps it out of command history and ps output.

• mysql -u root # Log into MySQL, which doesn’t yet have a root password.
• Delete the non-localhost root account.
• Set passwords for root@127.0.0.1 & root@localhost.
• Log out of mysql:

[root@mmm ~]# mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.0.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> drop user 'root'@'mmm.reppep.com';
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@127.0.0.1 = password('unencryptedpassword');
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@localhost = password('unencryptedpassword');
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye
[root@mmm ~]# mysql -u root
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
[root@mmm ~]# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.0.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> exit;
Bye
[root@mmm ~]# 

Create a MySQL Database & Account for Movable Type

• Create a movabletype database http://dev.mysql.com/doc/refman/5.0/en/create-database.html & movabletype user with access to it http://dev.mysql.com/doc/refman/5.0/en/grant.html.

mysql> create database movabletype;
Query OK, 1 row affected (0.01 sec)

mysql> grant all on movabletype.* to movabletype@127.0.0.1 identified by 'unencryptedpassword';
Query OK, 0 rows affected (0.00 sec)

Backups

I’m not covering MySQL backups here, but I use http://sourceforge.net/projects/automysqlbackup/.

Red Hat Kickstart is the right way to install RHEL, but Red Hat’s Linux documentation assumes the host can boot via DHCP. If DHCP (or bootp) isn’t available, there are some obscure options to provide the network configuration on the linux command line (at the boot: prompt). These options are ip, netmask, gateway, and dns, as in:

linux ks=http://10.10.10.100/kickstart.cfg ip=10.10.10.101 netmask=255.255.255.0 gateway=10.10.10.1 dns=10.10.10.100

I found several pages of kickstart docs which are missing these options, but they are listed on http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Installation_Guide-en-US/s1-kickstart2-startinginstall.html.

I’ve been a serious BBEdit user for years (I suspect a beta password many years ago was a joke at my expense). I use vi daily but am not a fan, and I find emacs inexplicable. As an system administrator, editing text files (typically configuration files and scripts) is a large part of my job.

So spending my days using Windows, with no BBEdit, was a concern.

I used kate a bit, and it’s not bad, but it’s limited (cannot even compare 2 windows!), and Exceed’s Copy & Paste support is extremely erratic.

Many people suggested UltraEdit, but due to an installer issue, I was unable to use it. That issue has been fixed, and I got it running today. I’m quite impressed, although I have already discovered that its sorting capability is downright feeble compared to BBEdit’s GREP-enhanced sorting. I see that UltraEdit offers 4 different flavors of Find & Replace: Plain, perl regex, UNIX regex, and UltraEdit regex. This seems crazy to me — I consider anything that’s not 100% backward-compatible with PCRE a bug, but I am not selling to a population of users who live in MS Word.

Truly weak: Find for $ doesn’t work properly in perl regex mode with UNIX line endings.

I am both impressed and mildly aggravated. UltraEdit is much better than anything else I’ve used on Windows (or Linux or Solaris), but it’s also less polished than BBEdit. I’m not sure how much of my frustration is because I have the advantage of years of experience of BBEdit — compared to only a few hours to learn UE so far — and how much is real deficiencies and lack of polish in UE.

“Give me a place to stand and a lever long enough and I will move the world.”

I was grumbling under my breath at a configuration management system today, and reminded of this wonderful statement by Archimedes.

Configuration management is the discipline of building systems which manage other systems — cfengine is a well-known open source example. I needed to reboot a few hosts on a regular schedule — easily handled in 5 minutes with “vi /etc/crontab” on each, or an ssh loop to append to the same file on each affected system. I was struck by how many levels of indirection I needed to traverse to get this done with configuration management. This in turn prompted some thought about why jumping through the various hoops was worthwhile.

There are many excellent reasons to use configuration management:

• Time savings — over repeating the same actions over and over; this increases with the number of hosts involved.
• Consistency — configuration management ensures that (portions of) systems which should be identical really are.
• Reproducibility — because CMS is naturally tied into version control, it is easy to either examine or recreate the state of affairs at an arbitrary time in the past.
• Modeling — a CMS ends encompasses a representation of all the systems it manages. This efficient representation of those systems is quite useful for examining and comparing them. It’s especially useful with a large or dynamic population of administrators, as it provides a single place to learn about the whole constellation of systems, and enforces some consistency among the various ways admins can manage systems.

In the simplest case, to make a machine reboot once, I could pull the plug and put it back (assuming I was near, or could get to, the machine). In a non-CMS scenario, I would do it with ssh and the shutdown -r. In this case, it was considerably more involved:

• Launch PuTTY.
• Log into a system with a checkout of the CMS configuration files.
• Find the appropriate file (non-trivial if the managed constellation is complicated).
• Fetch the latest version of the file (with multiple users, it’s unlikely my checkout is current).
• Edit the file corresponding to /etc/crontab or /var/spool/cron/root (I used kate, as I don’t enjoy either vi or emacs, and BBEdit wasn’t available); kate popped back an X11 session tunneled through ssh.
• Create a pair of local machine sets in the file (cfengine calls these ‘aliases’), each including half the covered systems (the systems reboot at staggered times, so they’re not all down at once).
• Create the pair of crontab lines, one for each machine set, embedding the pair of different reboot times and the shutdown -r command.
• Check the modified crontab file back into the version control system; enter a message for the change log.
• In a distributed CMS, staging hosts pick up the changes from version control, either on a schedule or when manually kicked for emergency/rush changes.
• The affected hosts pick up the change from the CMS, and implement the specified change.

The reason Archimedes’ quote is apropos is that configuration management provides excellent leverage — I can edit one file in one place, and easily affect several systems (potentially hundreds or thousands). Each hoop I have to jump through provides an additional fulcrum. I can sit at my desk and use PuTTY to log into dozens of systems, across the world — without even knowing where they are. Each change I make to the version control system is automatically picked up by every host participating in the system, and available to every admin with a checkout. I don’t have to log into 8 machines (even uninteractively) to make them reboot — I can orchestrate it all from my local workstation.

Unfortunately, mistakes are leveraged too; there is often no good way to test changes to production systems during business hours. If the changes are restricted to non-production hours, when the admin might not be around to monitor them (and shouldn’t have to — it’s an automated system, after all!), the window could be closed by the time the admin sees whether the change was successful. Missing a change window can easily defer a change 24 hours.

I needed to install the Citrix ICA client on CentOS 5.2 (RHEL 5.2), but it has very strange dependencies — it complains about a version of libXaw which is present, demands an older version of libXm, and requires manual installation of openmotif 2.2.

The trick (thanks, FriedChips!) was yum --nogpgcheck localinstall ICAClient-10.6-1.i386.rpm, rather than rpm -Uvh yum ICAClient-10.6-1.i386.rpm. This way yum chased the dependencies for me, and didn’t refuse to install the unsigned Citrix package.

Next I associated launch.jsp with /usr/lib/ICAClient/wfica.sh — Citrix should have used .ica instead, because .jsp is used for other things. IIRC, EMC NetWorker used .jsp to launch their graphical console.

Unfortunately the ICA client insists on being wider than the physical display, but I can work around that. I wonder if it’s because I simultaneously connected to the same XP system via RDP from both Linux and a Mac with different resolutions.

Update: Citrix is fixed on the size of my MBP’s 1440900 main display, which means it doesn’t fit properly on the MBP’s external 12801024 (or landscape 10241280) or my Linux box’s 12801024.

Annoyingly, Citrix assigns the Mac’s Command key to Alt on the Windows host. This doesn’t work well, because although they avoid most Command key combinations in the ICA Client, Command-Tab switches Mac apps rather than Windows windows. Guys, just use the Option key! It even says alt on it, and nobody needs that key for Mac specific functions! Today’s happy discovery: Command-Option-Tab switches Windows apps.

Next I have to figure out how to de-assign Alt-Tab from switching virtual workspaces in KDE. Copy & Paste don’t work consistently when connected from KDE either, presumably because some events are being interecpted locally and others are being passed through. I won’t need to use KDE as a Citrix terminal for much longer, though.

Crud. After all that, the Citrix ICA client doesn’t display most text, making it useless. I can get some things to display by selecting them, but many things (including dialog boxes) are un-selectable. Junk!

Ouch! At 10:31pm last night, I started patching both Linux servers running reppep and associated domains, prompted by Rich’s BIND alert. At 12:33am, www.reppep.com finished installing approximately 255 CentOS patches (including BIND), and I rebooted. Everything looked fine, and I went to bed. This morning, I thought it a bit odd that I didn’t have any new email, but not that unusual.

Melissa left me a message that mail wasn’t flowing, but I couldn’t fix it at work. Tonight I discovered that amavisd-new, which handles filtering for reppep email, was unable to start. Strangely, it was complaining about the Compress::Zlib perl module, which was actually installed (version 2.008, via the perl-Compress-Zlib-1.42-1.fc6 RPM). Some more digging indicated Scalar-List-Utils-1.19 needed to be reinstalled, which enabled amavisd-new to start (it checks for Compress::Zlib and refuses to start if it finds something wrong, which was apparently triggered by the Scalar-List-Utils issue).

mailq showed me postfix was now getting errors from amavisd-new about MIME::Parser and File::Temp. CPAN reinstalled MIME::Parser and said File::Temp was already current.

I bounced amavisd-new again, and tried postfix flush. Over the past 15 minutes, postfix has delivered the ~~650 outstanding messages, and all seems well.

Separately, Alex noticed our blogs were inaccessible, but bouncing BIND tonight cleared that — odd, as I checked http://www.bertpepper.com/ and got valid DNS resolution from both nameservers immediately after patching, but obviously something I didn’t notice was still scrambled.

Anyway, at 8:45pm, all seems present and correct.

Sorry for the disruption!

I use Handbrake to rip DVDs for my iPhone and Apple TV (and previously for my Treo 650, at much lower quality). It’s excellent, but takes a long time, and slows down the whole system while compressing. My PowerBook G4 wasn’t powerful enough to do H.264 compression in reasonable time, so I got into the habit of ripping the DVDs on the Mac with Mac The Ripper, copying them to the server, converting on the server, and copying back down to the iPhone through iTunes on the laptop. Transferring a full DVD via gigabit Ethernet takes under 10 minutes, which is why we have 3 1/2 gigabit Ethernet switches (8-port, a pair of 5-ports, and the 3 ports built into the Time Capsule) in a 3 1/2 person (2 grown-ups, a 5-year-old, and the cat) home.

Since then I’ve upgraded the laptop and replaced the server, so they’re now both 2.4GHz Core Duos, but I still prefer to avoid the load on my personal machine — especially now that we have an Apple TV, which likes large high-quality video files. The MacBook Pro runs HandBrake.app (Mac GUI), while the server runs HandBrakeCLI (Linux).

HandBrakeCLI is designed to convert individual files and is awkward for converting whole DVDs. Fortunately I found MediaForkCLI-batch.sh, a wrapper which is good at converting all the real titles from one or more DVDs; it automatically ignores short titles such as previews, and titles that have already been transcoded. Unfortunately MediaForkCLI-batch.sh is languishing, and hasn’t been updated for HandBrake v0.92, but it was straightforward to update it to the new syntax.

In the process I made some changes and added an alternate version with different settings optimized for our Apple TV. On the Linux box (named inspector), the scripts are ~/bin/appletv.sh & ~/bin/iphone.sh. I put newly ripped files into ~/tivo-inspector/input/ on inspector, and these scripts rip whatever they find there, generating files in ~/tivo-inspector/output/.

On the MacBook Pro, I keep video files in ~/tivo, and ~/tivo/tivo-inspector is a symbolic link to tivo-inspector on inspector, the server — mounted via AFP. This makes it easy to move files back and forth.

For convenience, I have a couple aliases on the MacBook Pro which log into the Linux machine, start a screen session, and run the appropriate script. This means that if I disconnect the laptop, I can later ssh back into the server and use screen -DR to reconnect to my running conversion session. When the conversion is done, the aliases open the remote folder in the Finder. I move the files back up out of input & output, drop them onto iTunes, and then either Sync the Apple TV (which gets copies of my whole iTunes Library), or sync them to the iPhone.

The scripts are appletv.sh & iphone.sh.

If you use these scripts, you should carefully review the encoding settings. In particular, I encode Apple TV audio to AAC stereo, because our Apple TV is connected to a pair of stereo speakers with a subwoofer; they cannot handle Dolby Pro Logic, which is better for most TVs.

You will also have to update paths in the scripts — to the input and output directories, and to HandBrakeCLI itself — to make them work on your own system.

Here are the aliases in sh/bash format; change the equal signs to spaces for csh/tcsh:

alias appletv="ssh -t inspectore time screen bin/appletv.sh; open ~/tivo/tivo-inspector"
alias  iphone="ssh -t inspectore time screen bin/iphone.sh;  open ~/tivo/tivo-inspector"

I got Exceed, and after reassigning my left modifiers to X instead of Windows, kate is quite reasonable. It uses kompare for graphical diff, and comes with some CVS plugins. I prefer BBEdit’s diff display (although BBEdit’s diff has been broken for years). I’m not sure how I managed to view a couple windows in xemacs from kompare, but I can probably avoid that in the future…

kate is clunkier and less featureful, and not as configurable as I expect — the commands I want to assign to the toolbar, for instance, are not available in that context. On the other hand, BBEdit doesn’t use a toolbar at all, and rearranging menus is only supported in limited ways, so I’m not convinced kate is inferior here — it may just feel like that to me as a BBEdit user. Hopefully Subversion support is available for kate, but that doesn’t actually matter to me right now.

I need to get Copy & Paste working between Windows (including PuTTY) and Exceed; hopefully this will be straighforward, but it doesn’t just work.

Per IDM, UltraEdit cannot be installed without admin rights, which I do not expect to get, so that’s out — at least until they offer an alternate installer.

kate icons are a bit fuzzy, but they fit the Linux aesthetic, and the fonts are very nice.

After DreamHost’s breach 8 months ago, I was aggravated at their poor handling of the situation, but willing to give them the benefit of the doubt, and still happy with their low prices and flexible services.

With the new bad news and worse confirmation (still with poor incident handling), though, it’s time to get out of dodge.

I have moved Extra Pepperoni back onto my own hardware. I started blogging on Apple’s Blojsom install, but gave up on Tiger Server for Blojsom (and Mailman) because the services kept silently shutting down, leaving me to notice they were disabled days or weeks later (no fault of Blojsom or Mailman — Apple didn’t do a good job porting SpamAssassin either). Bringing up a WordPress blog and mailing lists at DreamHost was easy and cheap, but that’s no good if they are unsafe.

I’ll look at moving a couple very light-duty Mailman lists off DH next, but the lists are so lightly used I’m not too concerned. There just isn’t any confidential information on the mailing lists, aside from their tiny subscriber lists.

Ah, well. I now know much more about WordPress and MySQL than I cared too, but the setup wasn’t too bad. I hadn’t realized how many customizations and tweaks I made to WordPress until it came time to recreate them on my own system:

1. Almost Spring theme (included by DreamHost); with minor hack
2. PHP Markdown Extra; with minor hack
3. MySQL admin UI
4. WP-DB-Backup (DH included one, which I’m no longer using)
5. mod_rewrite for permalinks
6. Admin-SSL, with “Shared SSL” tweak, integrated into my existing SSL site (meaning EP is available through two different “sites”, and I have to keep the Apache configurations reconciled)
7. Twitter
8. WP-Cache (DH standard)
9. Akismet anti-spam registration
10. Technorati pinger (came over automatically with the DB).
11. Fix for widget.php to use legal JavaScript tag.

It feels very very strange to be unemployed — it’s been 7 years since the last time, and I was too freaked out at Shooting Gallery laying me off to feel this way. Now that I’m a grown-up (having kid(s) means you’re responsible, even when you’re irresponsible!) it’s a good thing that we’re covered by RU insurance past the start date for GS insurance, but the whole experience is still very odd. I wiped the third computer today at 5:30pm, and am copying data off computer #4 (old reppep.com) right now in preparation for retiring it (it’s falling apart, apparently — optical drive died an hour ago).

Now I just need Apple to update the MBP15s, so I can replace this PowerBook. It’s doing better than I thought, though — doesn’t seem any doubt that it will serve until the next update.

RU IT did right by me today — a grand spread, consisting of John’s pizza, baby back ribs, beef ribs (they looked like something from The Flintstones), and chicken wings. A nice (short) speech by Armand, and well wishes all around. Elaine hung a bunch of colorful signs, which delighted Julia.

I closed out my helpdesk tickets, turned in my keys (forgot to turn in my ID/swipe card, though), and updated the documentation on our load balancers again, as well as re-re-recapping for my co-workers. I had to say “Look, when you feel like you’re an idiot, don’t worry — I felt like that repeatedly for years while working with these. The Big-IPs are absurdly complicated. Two kernels, a super ’switch card’ that’s doing all kinds of crazy (non-switch) stuff, over 20 IP addresses, 8 networks, plenty of bugs, and delays in getting technical support. It’s not you!”

Maybe I’ll have some time to investigate Linux & Windows text editors.

On Feb 19, 2008, I shut down the old reppep.com server, which ran Mac OS X 10.4 “Tiger” Server, and replaced it with a new (cheaper and faster) PC running Linux. Unfortunately, the password formats are incompatible, so I apologize to app reppep users for the disruption.

Please call me if you have an account on reppep.com and haven’t received your password already, or find anything not working right.

I switched from Apple’s jabberd to Openfire, which doesn’t use the UNIX system accounts, so let me know if you want a chat account (compatible with iChat & GTalk).

[Done] I forgot SquirrelMail address books — should be able to bring those over too.

• Firewall problem fixed. SMTP MX issue fixed.
• Virus filtering problem fixed.
• Webmail certificate fixed.
• Quota problem fixed.
• Virtual domains for email fixed.

As of 5pm, I don’t know anything that doesn’t work (aside from SquirrelMail address books) [fixed Thursday].

Thanks for your patience!

As of 10:30 on the 20th, things seem to be working. Something’s screwy with amavisd-new’s quarantine, but mail is going through. I reinstalled Openfire, and chat seems okay under the correct hostname/certificate name now (will try signing it as ca.reppep.com later).

Good timing — the optical drive on the old server died tonight.

I have distributed all the new temporary passwords, so any users having trouble logging in should let me know.

Markdown.cgi is still broken, but I’m the only person who uses it here, so I’ll get to it.

On Thursday the 21st, I found a problem with amavisd-new — it had quarantined 32,000 messages in a single directory, and was stuck (apparently ext3 doesn’t support more than 32,000 files in a directory). I cleared it out and finally managed to disable quarantine, which wasn’t as easy as it should have been, and the backlog of messages have been delivered as of 9:15pm.

At 11pm, I fixed an issue preventing SMTP AUTH from working properly, which was interfering with sending email to non-reppep addresses.

I was quite impressed by Joel’s description of the hiring process, and we’ve been doing a lot of interviewing for System Admins lately. I put together a list of standard questions to ask during interviews, which has been quite helpful in judging a) how much technical knowledge people have, and b) (just as important) how good a match they are for the skills void we were trying to fill at the time. Here they are, for the next person who needs to perform a similar exercise.

1. How many systems does your team manage (Linux, Solaris, Windows, etc.)?
2. How large is your team?
3. Which OS are you most comfortable/familiar with?
4. Which Linux flavors are you most comfortable/familiar with?
5. Which Red Hat versions are you familiar with?
6. Are you familiar with kernel programming or configuration?
7. Have you done any custom packaging or kickstarting?
8. Have you used or managed Sun JumpStart?
9. How much experience do you have with Sendmail?
10. … NetWorker? Version? Managing backups, or just configuring clients?
11. … LDAP? Brand & version? LDIF or just querying?
12. … firewalls (iptables, ipf, etc.)?
13. … network administration (Cisco, sniffing, etc.)?
14. … Apache httpd?
15. … Tomcat & Java?
16. … EMC (Clariion, PowerPath)?
17. … shell scripting, and with which shells?
18. … perl scripting?
19. … Veritas VM/FS? Versions?
20. … Veritas Cluster, or other HA? Versions?
21. … snapshots? In which products?
22. … load balancing
23. … Oracle (as SA, not DBA)?
24. … HPC?
25. Please briefly explain the difference between RAID 1 and 5. What are layered RAID levels, and when are they appropriate?
26. What sizable projects have you done recently?
27. Why are you leaving your current employer / did you leave your last employer?
28. Please give specific examples of some routine tasks you’ve performed recently.
29. Have you done systems specification and design (servers, multi-server configurations)?
30. Have you worked with customers directly, or primarily with/for other IT personnel?

It didn’t make sense to publish a list of questions when I was involved in the interviewing process, but now that I’m leaving Rockefeller and no longer interviewing UNIX Admins for them, I can post my sample questions.

For several years, I’ve been saying Apple made a bad choice when they picked Cyrus IMAPd as the POP/IMAP server for Mac OS X Server. It’s a huge and complicated system, encompassing IMAP, POP, SSL, Sieve filtering, LMTP delivery, USENET news, clustering/proxy (Murder), pluggable authentication (SASL), etc. I cannot think of a single company outside Cupertino where it would make sense to run an enterprise mail system on Mac OS X Server, but Apple continues to add these inexplicable high-end features to its mail server, most recently XSan-based email clustering in Leopard Server.

The statement that convinced me (shortly after I had migrated to Cyrus IMAPd on Mac OS X Server 10.4 “Tiger”) that I would never choose to run Cyrus for my personal use, was the following — which I came across again today:

Installation Overview

This system should be expected to have the same order-of-magnitude installation complexity as a netnews system. Maintenance should have similar complexity, except administrators will have to deal with creation and deletion of users and will have the option of managing quotas and access control lists.

USENET news is infamously demanding and bandwidth intensive. It would be wonderful if Apple had taken Cyrus IMAPd, repackaged it (without too many changes!), and put a powerful and simple interface on top. The did this quite successfully with Apache httpd (although Server Admin breaks down on complicated configurations and has obscure bugs). Lots of people use Mac OS X Server to run websites and think it’s easy & simple. Considering the typical reactions of those same people to the httpd .conf files “under the hood”, this is a noteworthy triumph. Similarly, Time Machine provides a reasonable approximation of scheduled snapshots on a high-end NAS for do-it-yourself file recovery, with a simple interface that insulates users from the nitty-gritty of copy-on-write and hard links.

Cyrus did not get as much attention, though. Basically, Apple makes it pretty easy to create email accounts, provides a Repair button for the overall Cyrus database, and provides a Reconstruct button for individual accounts. That’s about it. Unfortunately, Apple doesn’t really document maintenance beyond “press the button and it will fix your problem”. I’ve had several serious database problems which Apple’s Repair button did not help with. Those were bad times.

Similarly, I have had problems where users could not log in, but Workgroup Manager claimed their accounts were usable. I eventually discovered that resetting passwords with passwd works sometimes, and re-setting passwords in Workgroup Manager works consistently, but when I asked Apple about it, the eventual response was basically, “Yes, that’s bad; you should restore your accounts from your recent Open Directory export.” Not a good answer.

It doesn’t help that Apple’s SpamAssassin and ClamAV installations are broken, as these result in more spam and slower deliveries.

So why am I planning to migrate to Cyrus IMAPd on CentOS 5.1? Well, I’d really like to just copy my 5gb mail directory to the new system and have my clients not notice the difference. Eudora doesn’t handle (IMAP) change well — renaming a single IMAP directory can force it to download all messages again, and various other things can cause Eudora to lose date stamps on sent mail, or message state information (when it gets disassociated from the actual message on the IMAP server). If I can make Cyrus work, I’ll be very happy, and if I can’t I’ll try Dovecot (Red Hat’s default) or Courier (which I hear is also good).

Also, I know it can work, and I have a rough model to work from on my Tiger Server, but if I wasn’t using Cyrus already I would stay away from it, as I wish Apple had done.

I’m digging deeper into CentOS (basically a free version of Red Hat Enterprise Linux) v5.1, and for me that entails dozens of runs through the installer, testing out kickstart configuration variations.

This led me to wonder if it is faster to install via NFS or HTTP. I couldn’t find a useful answer online, so I ran a couple simple tests. My client is a 2.4GHz Dell PowerEdge 600SC using SATA disks on a Promise TX4. My server is a dual 1.25GHz Power Mac G4, running Mac OS X Server 10.4.11. They’re connected via private network, using a NetGear gigabit Ethernet switch.

For NFS installations, anaconda takes a directory containing a DVD ISO (or set of CD ISOs), and automatically loopback mounts them as part of the installation process. This is very handy with the CD ISOs, as it doesn’t require much configuration on the server — just an NFS export.

In contrast, HTTP installation doesn’t work against ISOs — the web server must serve up the individual files, whether from a loopback mount on the server, or a directory where the files have been extracted. With the 6 CD ISO files, this is quite a nuisance; with the DVD ISO, it’s not so bad.

My fairly complete kickstart configuration installs 2,103mb of packages. Installation times were quite similar, but a bit faster for HTTP, at 18:28 for package installation and 24:37 total. Via NFS packages took 19:57; total was 27:38.

For reference, I used the following partitioning configuration, which factors into total installation time:

part /boot --onpart=sda1 --fstype=ext2
part /     --onpart=sda2 --fstype=ext3
part swap  --onpart=sda3 --size=2048
part /var5  --onpart=sda5 --fstype=ext3
part /home6 --onpart=sda6 --noformat
part /sdb1  --onpart=sdb1 --noformat

The partitions already existed:

[root@pe ~]# df -hl|grep -v tmp
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             7.7G  2.8G  4.5G  39% /
/dev/sda6             664G  198M  629G   1% /home6
/dev/sda5             3.9G   73M  3.7G   2% /var5
/dev/sdb1             688G  198M  653G   1% /sdb1
/dev/sda1             251M  7.4M  231M   4% /boot

I used the following set of packages for testing:

%packages
@core
@base
@editors
@system-tools
@text-internet
@dns-server
@base-x
@network-server
@games
@web-server
@mail-server
@server-cfg
@admin-tools
@development-tools
@graphical-internet
@office
@development-libs
@x-software-development
@gnome-desktop
@graphics
@kde-desktop
@sound-and-video
kdegraphics
kdemultimedia
-bluez-gnome
-bluez-hcidump
-bluez-utils
-evolution
-evolution-connector
-evolution-webcal
-ibmasm
-krb5-auth-dialog
-mod_perl
-mod_python
-numactl
-openldap-clients
-openoffice.org-calc
-openoffice.org-draw
-openoffice.org-graphicfilter
-openoffice.org-impress
-openoffice.org-math
-openoffice.org-writer
-openoffice.org-xsltfilter
-pam_krb5
-pcmciautils
-php-ldap
-planner
-rp-pppoe
-sabayon
-sendmail
-sendmail
-sendmail-cf
-squid
-stunnel
-sysreport
-system-config-samba
-tux
-vconfig
-ypbind

In Leopard, Tab completion in bash doesn’t immediately append trailing slashes to symbolic links that point to directories. When I complained about this change to Apple, I was told it was user configurable, and I should just configure the old behavior. It took me a while to actually find the solution — partially because it isn’t within bash itself, and partially because it was quite a nuisance but not a serious problem.

For example, ~/www is a symlink to /Volumes/www, and I cd to directories below it quite frequently. I’m in the habit of typing “cd w[Tab]/pu[Tab]“, which should expand to “cd www/public_html“. This broke in Leopard — I needed an extra Tab to get the / — otherwise I’d end up with the ugly and non-functional “cd wwwpu“. Anyway, the fix is:

echo "set mark-symlinked-directories on" >> ~/.inputrc

Now Tab completion works the way I want it to again. Thanks, stylishpants!

Red Hat has, with good consideration and foresight, been pushing people to use logical volume management for a while. It’s not completely integrated into the RHEL5 installer, but they’re pushing hard to make it ubiquitous, and telling people this is the right way to do things. Unfortunately, the syntax for specifying logical volumes within DOS-style partitions is still a bit obscure, and the manual page examples don’t show the LV syntax; this is fixable, but will take time.

I used software RAID and LVM on my new installation, but it doesn’t boot — I’ve found several articles on making GRUB work with software RAID, so I believe I’ll be able to get it working. The docs say I should be able to just use “lvm” (which is present) to get an lvm shell, but neither lvm nor lvm.static does anything — they just dump me back in bash.

Fortunately, “linux rescue” finds my partitions (this time), but not being able to even list out physical volumes is worrisome.

I want mirrored /boot, but it’s RHEL’s mirrored /boot capabilities are pretty limited:

If you are making a RAID partition of /boot/, you must choose RAID level 1, and it must use one of the first two drives (IDE first, SCSI second). If you are not creating a seperate RAID partition of /boot/, and you are making a RAID partition for the root file system (/), it must be RAID level 1 and must use one of the first two drives (IDE first, SCSI second).

Speaking of LVM being immature, check out What is the process to fsck lvm volumes? in the Red Hat Knowledgebase:

First, boot into rescue mode by using the correct media. This is very important: When prompted to mounted the drives, do not. Using fsck on a mounted filesystem will destroy all the data on that file system. This is unrecoverable. The data will be gone forever–save for very expensive hardware-level data recovery.

For over a year now, I’ve been following the development of Mac OS X Server 10.5 Leopard and testing betas, and anticipating upgrading reppep.com from Tiger Server on a dual 1.25GHz Power Mac G4 to Leopard Server on a dual 2GHz Power Mac G5. Over the weekend I had a change of plans, though.

Although I support Mac OS X Server at Rockefeller, I don’t recommend it for most requirements, as Linux compares favorably for transparency (some of the MOSXS internals are unique and poorly documented), server software compatibility (although Macs are quite good here too), and price/features at the low end. A Core Duo Mac mini has plenty of juice to saturate our 768kbps/3mbps DSL circuit, but adding a couple drives more than doubles its price, and Apple’s software RAID is quite broken; Linux software RAID is apparently quite good; I might eventually switch to hardware RAID. An Xserve is a great piece of hardware, but it’s a bit exotic and I can get a fast generic PC cheaper; I don’t want all the high-end features for a box that sits in our apartment.

Additionally, I’ve read perhaps 600 pages of docs on Leopard Server, and had at another 400-1500 yet to go. This is an investment I was finding hard to justify. The migration process is quite complicated, and Apple doesn’t support migrating accounts from a Tiger system to a Leopard system — I don’t want to do an upgrade. I could clone the G4 to the G5 and upgrade it there, but I prefer to handle upgrades as scratch installations with manual migration of applications, so I know exactly what’s been done. A lot of this is masked by upgrade procedures.

As part of this, I’ve decided to invest a bit more time in learning RHEL5 — we have a couple systems at Rockefeller, but not much in production yet, and now seems like a good time to dig in some more.

Fortunately, all the services I’ve been using on reppep.com are available on Linux (and FreeBSD), so aside from another incredibly inconvenient password change cycle (for which it is arguably time anyway), the switch should be largely transparent to reppep.com users, although I still have plenty of research to do.

A brief timeline of reppep.com

1. 1999: I left the National Audubon Society, and bought the Power Mac 7300 with accelerator card I’d been using there. I set it up with LinuxPPC and Apache, and started offering free web hosting to friends & family. LinuxPPC was eventually discontinued.
2. I upgraded from LinuxPPC to Yellow Dog Linux, which was better than LinuxPPC, but had serious flaws.
3. 2001: I was working on a couple remote FreeBSD machines (as admin of the Info-Mac server, and a user on the Apache Software Foundation userhost), and decided to learn more; I bought a cheap Celeron PC and installed FreeBSD 4.3 (IIRC); I upgraded through about v5.1 and a Pentium 4 (giving the Celeron box to the Info-Mac Archive, where it became the Info-Mac server for a while). I learned a lot about FreeBSD and UNIX in general, but eventually realized I was investing more time learning FreeBSD than I could justify. The best thing about FreeBSD is not a technical feature, but rather that the user community is so rich with knowledge. Reading the FreeBSD-STABLE list was amazing, as there was so much depth, freely shared with the community. While running on FreeBSD, I added mail services to the web services I had been offering. Note: Disruptions to personal email service are much worse than problems with personal web service.
4. 2005: It became clear that I needed anti-spam, so I began researching SpamAssassin. While I was figuring out how to build the SMTP sandwich, with a public untrusted Postfix listener on port 25 & 587, and a filter, and then a listener on a high port like 10025 to accept and deliver mail to actual users, I installed a beta of Mac OS X Server 10.4 “Tiger”, which had the whole thing implemented, plus ClamAV as a bonus. I started testing heavily before the release, and switched to MOSXS 10.4 shortly after it was finalized. It’s been very good, but as time has passed, I’ve had more and more problems. In particular, Apple chose to use Cyrus as an IMAP/POP server, and Cyrus is complicated, but Apple ignores the complexity; this can make troubleshooting impossible. The SpamAssassin installation is slightly broken; it’s a bit too old to offer the newer SpamAssassin self-upgrade mechanism. Server Admin is great, but has a bunch of bugs around SSL certificates, some of which destroy the certificates. Blojsom was nice, but Apple’s installation was very unstable; I eventually moved my blog to WordPress hosted externally.
5. 2008: I intend to switch to CentOS 5.1, which is basically a (legal) no-charge clone of Red Hat Enterprise Linux 5.1. This should make future upgrades a bit more straightforward, as I won’t have to deal with Apple’s Open Directory (OpenLDAP); it will also give me a bit more experience with RHEL5, which is a better investment for my time than Leopard Server.

Update: It’s sillier and sadder than I thought. See below.

Today (2007/11/12), Oracle announced Oracle VM, their free competitor to VMware and (Citrix) Xen. A few months ago, Oracle announced “Unbreakable Linux”, which is their re-branding of Red Hat Enterprise Linux. There are already many free Red Hat flavors, including CentOS, but not too many companies have built business models on attempts to take Red Hat support business away from Red Hat.

Oracle has. They made many loud claims of being cheaper and better than RHEL, while claiming this wasn’t an attack on Red Hat. Red Hat was pretty quiet about Oracle Linux, but did point out that Oracle’s claims to be actively fixing bugs in RHEL (supposedly faster than Red Hat does) without forking RHEL were impossible — as soon as there’s a fix which isn’t available from Red Hat, that’s a fork.

There’s been a lot of ill feeling both ways over this, but of course neither company is willing to publicly and unambiguously badmouth the other.

Today we see another step in Oracle’s (Linux) plan: Oracle VM is free, but Oracle offers paid support. The best part is this, though:

What is the difference between Oracle VM and the virtualization that comes bundled with Oracle Enterprise Linux?

As part of the Unbreakable Linux Support program, Oracle supports virtualization that is included with Oracle Enterprise Linux 5. Please note that Oracle products are not supported to run in that environment. Any customer who wants to deploy Oracle products in a virtual environment should use Oracle VM, and subscribe to Oracle VM support. Oracle customers should refer to MetaLink note 466538.1

Translation: We sell RHEL5 (which includes Xen as part of the base price) but we don’t like it, because we want you to pay more for Oracle VM instead. We cannot realistically either break or drop support for Xen, even though we’d really like to, but we do get to chose what “platforms” we support Oracle on, so we’ll support Xen, and Oracle on Linux, but not Oracle on Xen. Please don’t think too hard about that one. It makes our heads hurt!

Update 2007/11/13: I missed the fact that Oracle VM is based on Xen. This means Oracle wants to sell you “Unbreakable Linux”, but wants to charge an extra $500 to virtualize its own software on “Oracle’s” Linux platform. I thought they were claiming Oracle VM was better than RHEL’s VM, but that can’t stand even cursory scrutiny, given that they’re basically the same code. Additionally, their

• Three times greater efficiency than current x86 based server virtualization products;

has to be in relation to VMware which is not paravirtualized, but there is no way Oracle’s brand-new Xen build is significantly faster than Red Hat’s Xen kernel, running on Red Hat’s Linux distribution.

Given that Oracle now recommends RHEL + Xen (from Oracle) as a platform for running Oracle Database & Applications products, Oracle’s lack of support for running on RHEL + Xen (when purchased from Red Hat) looks — I was going to say even more absurd, but this can’t be an oversight, so it’s just transparent corporate greed.

I just finished RH401: “Red Hat Enterprise Deployment, Virtualization, and Systems Management”. It’s a 4-day course, given Tuesday-Friday of this week. The course is normally Monday-Thursday, with an assessment exam (EX401) on Friday. Had I known this, I probably would have taken the course with the exam — I’d like to have that certificate. There are 5 tests (including EX401) to earn the exalted title of “RHCA”, Red Hat Certified Architect.

The course covered several major areas:

• Net booting (PXE, DHCP, & TFTP)
• Kickstart (automated installation of RHEL)
• Red Hat Network (rhn.redhat.com, a service hosted by Red Hat), Satellite Server (a local version of the service, which includes and installs net boot services), and Proxy server (a customized caching webserver which saves bandwidth and download time — a subset of the full Satellite)
• Building RPMs
• Xen virtualization

Xen is very cool — it’s perhaps halfway between VMware and Solaris zones (containers), so more efficient than VMware but less than zones. Xen offers live migration between servers and supports RHEL 4.5 as a guest OS. With appropriate hardware (preferably recent Intel or AMD CPUs with hypervisor instructions), Xen can also virtualize Windows and earlier versions of RHEL. VMware is much more mature, but very expensive (easily more than the hardware it runs on for standard 2-socket systems), so this was a useful preview, even if we don’t expect to use Xen much during the next year — perhaps for Rockefeller’s multi-user webserver, where we would like more isolation between users.

I was really there, however, to find out how to build custom RPMs for Rockefeller, manage them with custom RHN channels, and kickstart from a net boot server to streamline and automate installations.

Unfortunately this turns out to be surprisingly expensive, compared to what we pay to run RHEL. We normally pay $50/host/year for RHEL Academic Server, which is basically the Update & Management entitlements. This enables us to download patches from rhn.redhat.com (Update), and do a little bit more advanced stuff such as group systems in the RHN website (Management).

To use all the custom channels and kickstarting discussed in the class, we need a Red Hat Satellite Server (which costs about as much as all our RHEL Academic seats combined), and a $96 RHN Provisioning add-on Entitlement for each server. Combined, these would quadruple the amount we pay Red Hat annually for our servers, and I’m not at all convinced it would be a worthwhile investment.

We may instead get a Red Hat Proxy Server, which provides custom channels and costs much less than the full Satellite, and build our own kickstart server, forgoing all the Satellite features. This would be a shame, but might turn out to be the best compromise.

Another problem is that the RHN/Satellite back-end is RHEL4AS only — it doesn’t run on RHEL5, and it doesn’t coexist well with any other services. This is a larger Red Hat problem, rather than specific to the class, but it meant the class was a mixture of RHEL4 and RHEL5, and made things more complicated.

It’s enough to make one seriously consider CentOS, is a rebranded free version of RHEL. We don’t want to do that, though.

Paul, our instructor, was full of excellent tips on better ways to work with RHEL. Unfortunately, I avoid many of these (decidedly useful) techniques, since they only work on Linux (or only RHEL), and I generally stick to things common to Linux, Solaris, and Mac OS X. The neat stuff Red Hat has added recently, which he was excited about, would make my RHEL work more efficient at the expense of having to keep track of the RHEL way and the non-RHEL way. Those commonalities are essential for me.

Still, I learned a lot of useful stuff about RHEL, and now just need a chunk of time to set up a kickstart server and decide how to do DHCP — our DHCP scopes are managed by the Network Group, and we need a way to set up and manipulate kickstarting without asking them to make multiple DHCP & VLAN changes. I have some ideas for how to automate and customize the kickstart process, which I’d really like to test and implement.

I passed the RHCE exam, hooray!

This was much easier than becoming Dr. Pepper, and much easier & safer than becoming Sgt. Pepper.