Extra Pepperoni

I’ve said it before, and I’ll say it again: Apple should’ve picked a simpler mail server for Mac OS X Server — preferably one that used user-accessible mail files or folders, like UW or Dovecot or Courier (I’ve only used UW, and it wasn’t great, but it was simple, and Apple obviously liked plain files over databases enough to customize Blojsom instead of WordPress).

Somebody in Cupertino decided it was a point of pride that MOSX must be able to handle a large-scale enterprise workload, and they picked software that’s cluster capable, but who would choose (very nice) Xserves to provide mail service for tens of thousands of users? Apple isn’t even actively pursuing this market, although they assure us clustered Xserves running Xsan over Xserve RAIDs would be fast. No way — in that market, lack of RAID controller redundancy is an instant deal-breaker, just as lack of redundant power was in pre-Intel Xserves.

If you’re going to provide an industrial-strength mail server (intended to be no harder to install and configure than a USENET newsfeed — yowch!) to Mac users, you need to provide more than 2 buttons for troubleshooting Reconstruct (account) and Repair (database). They have never solved a problem for me.

When these fail, Mac users are out in the cold. Apple’s documentation is grossly inadequate for troubleshooting, their fora are not helpful, and their build of Cyrus is just different enough to make things more even complicated than they need to be.

I’ve had database corruption about 3 times in the past, and eventually writhed my way out of it. This time, my $15 CyberPower UPS (which I liked, as it worked better than my $100 APC & Tripp-Lite UPSes) spontaneously expired Christmas Eve while we were at my parents’ house. Feh! When we got back home, it wouldn’t turn back on.

Apparently this happened at a poor time, and a message got eaten (INBOX #76641, to be exact). Eudora simply couldn’t check my INBOX — the server died every time. Eventually, I got an informative error from SquirrelMail which helped finger the bad message, copied another message to user/pepper/76641., fixed up ownership, and sacrificed a few Tamagotchi while running /usr/bin/cyrus/bin/reconstruct (I’d love to know who decided on /usr/bin/cyrus/bin) with different guessed arguments, until Eudora started sucking down mail as it’s supposed to once again.

That was thoroughly unpleasant. Why can’t Cyrus survive a missing message? Why can’t it warn about a missing message, or an unreadable/root-owned message?

PS-It doesn’t help that syslogd periodically stops logging mail traffic. There’s probably a better way to make syslogd work again, but rebooting does the trick. Unfortunately, this means I often have no logs of problems since it dies a little later…

Update: Apple just updated Reconstructing cyrus mailboxes in Mac OS X Server 10.3, 10.4, which provides correct usages for the (very picky) reconstruct command, but no explanation whatsoever.

I use vi daily, but much prefer BBEdit. The way I integrate them has evolved over time (see previous posts here, Useful subversion shell aliases, and BBEdit Gems (which appears to be down right now). In particular, I no longer configure BBEdit directly in ~/.subversion/config.

The new improved integration uses BBEdit whenever it’s available (I’m in front of the Mac), and falls back to the default (vi) when I’m connected via ssh.

First, I created ~/bin/edit.sh to hand off to BBEdit. I use edit.sh whenever BBEdit is appropriate:

#!/bin/sh
# Edit in BBEdit, for programs that don't support arguments in $EDITOR.
bbedit --wait --resume "$@"

Next, I configured my bash profile to prefer edit.sh when I’m not connected via ssh (which means when I am in front of the Mac or using ARD/VNC), as my EDITOR and PAGER. My profile doesn’t automatically determine whether to use bbdiff for Subversion, because I sometimes find it necessary to use non-BBEdit diff for Subversion (there are cases where the svn-to-bbedit handoff doesn’t work well, and I have ended up editing scratch files instead of the real files, for instance). Here’s the snippet that does this, from my profile:

if [[ ! $SSH_TTY ]]
then
  if [ -x ~/bin/edit.sh ]
    then
      export EDITOR=~/bin/edit.sh
    else export EDITOR=vi
  fi
  if [ -x /usr/bin/bbedit ]
    then export PAGER="col -b | bbedit --clean --view-top"
  fi
else export EDITOR=vi
fi # [[ ! $SSH_TTY ]]

In addition, I set up aliases for reviewing output from the svn command, based on Bob’s suggestions. I just copy and paste one or more lines from svn output to review changes in BBEdit:

alias  A='bbedit --wait'
alias AM='bbedit --wait'
alias  C='bbedit --wait'
alias  D='true'
alias  G='svn diff --diff-cmd bbdiff --extensions "--resume --wait"'
alias  I='true'
alias  M='svn diff --diff-cmd bbdiff --extensions "--resume --wait"'
alias  R='svn diff --diff-cmd bbdiff --extensions "--resume --wait"'
alias  U='bbedit --wait'

I update and then review status so often that I built my own TLA:

alias sus='svn update && svn status'

The following two commands provide “Subversion diff, with BBEdit” and “Subversion diff, no BBEdit”:

alias sdbb='svn diff --diff-cmd bbdiff --extensions "--resume --wait"'
alias sdnb='svn diff --diff-cmd diff -x -u'

When I’m way behind and have a lot of changes to review, I use a one-off command to review all the changes since I fell out of step. I also scan the email log messages, but this catches all the changes, skipping intermediate versions which have already been replaced and consolidating multiple edits to the same file. The command is something like:

svn diff --diff-cmd bbdiff --extensions "--resume --wait" -r1039

PS-I get the Subversion client with “fink install svn-client-ssl“.

I just posted a couple pieces on Rich’s site: HTTP Authentication: a Primer & When Community Is Bad: Community and Commerce — Don’t Cross the Streams!.

My 1.5GHz PBG4 has been painfully slow for the last month or so. While I’m game to replace it with a MacBook Pro (Core 2 Duo, baby!), we haven’t sold our old apartment yet, so new computers are verboten this month. Reasoning that a) it used to be reasonably snappy, and b) running a 100gb drive with 0-5gb free for a couple years could cause severe fragmentation, I copied almost everything off, repartitioned to get rid of my 10gb OS test partition (which has been very very useful, but is no longer needed for various reasons), and copied stuff back.

Unfortunately, there was a giant fly in my ointment. Since I keep an exact duplicate of my iTunes library on two other machines (via rsync), and the library was more than twice the size of my other OS & data files, I didn’t back up the music with SuperDuper! Instead I just skipped it, and copied it back from another machine after SuperDuper!ing the restore.

Thus begins my tale of iTunes woe. My 48gb 13k-song library was trashed. The iTunes DB appears to store and prefer FSIDs for the actual media files. An FSID (FSSpec?) is a file number, guaranteed to be unique for a file system (volume, partition, disk). Of course, when one recreates the file system and copies files back, they get new FSIDs. When I opened iTunes, it wouldn’t play music. I now believe this was because it was trying to start off by playing an ‘MP3′ entry from the Library (database) which actually pointed to a non-music file (presumably one which had picked up an FSID which previously belonged to that MP3 file).

iTunes has a subtle problem with ‘missing’ files. Although it can skip over missing songs while playing, finding the next song in sequence that is available and picking up there, when starting to play music, iTunes can’t get past a missing song. The solution (if you know what’s wrong, but unfortunately there is no warning or indication of the problem — iTunes just does nothing when exhorted to make music) is to play a file which is actually present and correct, after which things work okay. And then to use something like the excellent Super Remove Dead Tracks AppleScript from Doug’s AppleScripts for iTunes.

Making things more difficult, iTunes kept hanging while attempting to update my iPod (Spinning Pizza of Death), and stopping after I started playing one song (so there was more wrong than just the won’t-start behavior, but I don’t know what). I had no music to listen to while I worked on the problem! Combined with having no iTunes while SuperDuper! did its thing, I was already going a bit (more) batty (than usual).

After spending a totally annoying amount of time thinking the iTunes DB & preferences were completely corrupt, I realized that of my almost 13k “tracks”, a little over 5k were the correct files, but more than half were pointers to the wrong places. Note that they all looked correct in iTunes, which had all the playlist, rating, etc. info in the DB (ratings are only in the DB — they aren’t stored in the files, which irritates me every time I lose the DB — no, this wasn’t the first time).

After opening a bug report with Apple, sending an Apple System Profiler report, 3 samples, and a copy of my iTunes Library file, I figured out what was wrong. I used BBEdit to strip out all the bogus Locations from an iTunes Library export, which meant iTunes unfortunately completely ignored those files, but it did re-import all the ones with good paths. This gave me a 5k-song library with ratings. Next I dropped my whole music folder on iTunes and lett it import, scan for gapless playback, scan for volume level, and scan for album covers, I had a mostly-working iTunes again, with 8k+ unrated songs and holes in all my manual playlists. Fortunately, I was able to create Smart playlists ‘1′, ‘2′, ‘3′, ‘4′, & ‘5′ from the old Library (containing the bad FSIDs), which imported mostly successfully. After rating unrated songs according to those playlists and a bunch manually, I had a hundred unrated songs, which was much more reasonable.

YUCK!

Update, 2006/12/16: Feh. Some of the FSIDs ended up pointing to real sound files, which means after I dropped the whole folder onto iTunes, it had duplicate entries for those tracks (one pointing to the right file, and one pointing to the wrong file). Fortunately these were pretty easy to recognize, because I had pairs of one rated (bad file) and one unrated (’new’ file from the drop), but I had already ‘cleaned up’ a few of these before I realized that I still had bad links. Additionally, since iTunes is set to “Keep iTunes Music folder organized”, for a few files that pointed to the wrong MP3s, iTunes helpfully renamed and re-tagged those MP3s to match the MP3 it ‘thought’ they should be. This means I have a few unidentifiable files with the wrong names. For some I can tell which is right by playtime, but for others I can’t know which is which, and there’s no sign of which files are missing from the library.

So Friday I spent several hours in a machine room, dealing with a mulish array. Having failed to anticipate how much time this would take, I also failed to bring a jacket. We spent most of the time waiting for various people and things.

Back in the day, I used to read Wired to see what new words and phrases they’d come up with to describe geek life.

My own contribution, inspired by the hours spent in our machine room, waiting for a callback/reboot/answer/whatever (both yesterday and on other days):

“Going to the beach“: Moving from the “cold aisle” (in front of the equipment, where the keyboards and displays are, and where the air conditioning system dumps chilled air) to the “hot aisle” (where the hot air is exhausted from the backs of the servers, before being sucked back into the A/C system for another cycle).

Example: “I’m going to the beach for a few minutes, while we wait for a callback from someone with a clue why the tool failed.”

I’ve been using the customized Blojsom installation Apple provides in Mac OS X (Tiger) Server to host my blog and a few others. Unfortunately, it’s not stable. Last week, it died several times, and I gave up on waiting for Apple to fix the problems in Leopard Server (expected early 2007).

I’ve migrated the blog (including all old my posts) to WordPress, hosted by DreamHost, at www.extrapepperoni.com. As a bonus, WordPress includes many more (interesting) themes than Apple’s very small (and classy, but simple) set, and additional features which Apple doesn’t provide. Unfortunately, Apple’s customized Blojsom can’t take advantage of the many designs people have developed for mainstream Blojsom, and its plugin architecture is broken as well.

The main advantage of Blojsom is that all posts are simple (pairs of) text files, while WordPress is based on a MySQL database. Fortunately, WordPress/DreamHost hide much of the complexity.

In addition to my blog, I’m moving a couple mailing lists to DreamHost, again because Apple’s Mailman is unstable.

For a long time, I ignored the Apple Keychain (see earlier posts for more Keychain travails). I didn’t want to keep my passwords anywhere accessible without my intervention. What finally made me give up was the fact that Safari prompted me for my Keychain passphrase on every single flipping page with a text entry field. Eventually I gave up, and started unlocking the keychain. It’s very useful, but Apple (Safari) effectively forces many users to use the Keychain by making it so intrusive and unpleasant. Why isn’t there a “don’t bother me” option in that dialog?

Then when I started using SSHKeychain, the Apple Keychain became much more important to me, because it contained the passphrase for ssh private keys. I am an aggressive locker. When I leave the room, I lock the screen. I do this at home (and irritate Amy), and I do this when I leave the cube farm at work. As a result, I unlock the Mac frequently, with a longer-than-average password. It’s a minor nuisance to type one password more than hourly, but if I had to unlock the screensaver, at least one Apple keychain, and one or more ssh private keys, I wouldn’t be able to get any actual work done before I bought an Uzi.

With SSHKeychain, I discovered that Apple doesn’t support locking Apple keychain(s) when the screensaver locks. Now I know that a major reason for this is that things break when the keychain is locked. In particular, .Mac sync throws all kinds of hissy when it doesn’t have access to your .Mac password through a keychain (I’ve counted 5 different prompts for my Keychain password so different parts of .Mac sync can connect). That’s obnoxious, and bad security.

Today’s brokenness is related. If you don’t have your .Mac password in an unlocked keychain, it’s impossible to get a .Mac iChat certificate. Instead you get a bogus error pointing to the “Forgot password” page. I didn’t forget my password, you robotic clown! I just won’t give it to you for safekeeping (this is on a multi-user server I rarely use, and where I don’t want or need saved passwords). I tried entering the password directly into iChat (faster than getting it into the Keychain at that point), but again iChat’s Encryption Assisstant failed with a misleading error. As soon as I cached my .Mac password, the Encryption Assistant worked. Two bugs (not accepting manual password entry, and not using a password stored in iChat preferences) + a misleading error message + forcing the user to inferior security (cached password) in order to get a security feature (encrypted chat)!

How perverse is that? Don’t answer, please. You’ll set me off again.