Extra Pepperoni

To content | To menu | To search

Saturday, March 3 2007

The Apple Keychain is cool, but also strange and problematic

I'm writing about SSL/TLS on Mac OS X, so I've been playing around with certtool (a very nice text-based interactive tool for working with certificates, based around the Apple Keychain), and security (a more powerful tool for dealing with keychains in general).

I've discovered that not only can't certtool export the certificates (or keys) it generates (it only exports CSRs, which is important but insufficient), but "security export" doesn't work for me either. I can't tell if it's pilot error, but the documentation is sparse and inconsistent.

Needless to say, I left some work in Radar for Apple's doc folks and security programmers this week.

More confusing, when I generate a certificate with certtool it shows up as expected in Keychain Access, but I don't see the private key there. My understanding of certificates is that there must be a private key for every certificate (although one key might be used by several certificates), so where is it? Does every keychain have am embedded and invisible private key, which you never see? This has implications for people generating certificates, as they might not be able to disentangle and distribute them in the future, if multiple certificates are tied to the same secrete key. If I could get "security export" to work, I could see if it emits the key along with the cert, but so far no joy.

Just to make things more interesting(!), I have private and public keys in my keychains. They're not SSHKeychain items, which are identified as such (and really just each hold an encryption passphrase for a normal triple-DES encrypted OpenSSH private key).

So I suspect there is a 'bedrock' private key in each of my keychains, (probably automatically generated when the keychain is created), which I can't see or access directly. But I don't know! This would make sense, as the keychain items are all encrypted, so perhaps they are encrypted with the same private key the public keys are tied to. But keychain icons can be decrypted and re-encrypted with another key; certificates are permanently tied to their private key.

More bugs: When you open Keychain Access and Show Keychains, you select a keychain in the upper-left corner ("Keychains" list), and only see keys from that keychain. But if you type into the upper-right search box, Keychain Access switches to showing all matches across all keychains, while leaving one 'selected' in the upper-left -- except that selection no longer controls which keychain's items are displayed, so the meaning of the selection changes (goes away, actually).

But the cool part is this

Launch Keychain Access, and under its Keychain Access menu, select Certificate Assistant. The Certificate Assistant (actually a separate program) has explanations of private/public keys and certificates, and assistants for generating CSRs, self-signed keys, even your own CA -- all behind a helpful GUI. For bonus features, you can use it to review existing certificates, or even interrogate an HTTPS server for its cert dynamically and review the cert (similar to visiting the site in Safari and clicking the lock icon), and Apple provided some nice automation, so when you create a CA, it gives you a configuration file you can give to other people. If they open the file, the Certificate Assistant launches on the recipient's Mac, and walks you through sending a CSR back to the CA Mac. Crazy! It claims to have sent an email request for a certificate signing on my behalf, but I didn't confirm or see it happen, so I'm a bit confused (again). But it's very cool, no question.


It's very difficult to be clear when talking about keychains. The Apple Keychain is an API, system feature, and file format (and apparently a framework, distinct from the Keychain Manager, SecurityFoundation.framework, and SecurityInterface.framework).

Each user and installation of Mac OS (including Classic) may have one or more keychains. Keychain Access is Apple's application for managing both the Apple Keychain (including overarching per-user settings) and individual keychains (both personal and system-wide). The Keychain MenuExtra offers a subset of the features in Keychain Access.

SSHKeychain is a third-party application that manages ssh agents, apparently derived from the Gentoo project's keychain (a popular command-line tool), but tied into the Apple Keychain, which may contain SSHKeychain items. Confused yet?

Even when I am able to keep it all straight, it's very difficult to explain this stuff to users who don't know all the pieces (anyone who already knows all the pieces probably doesn't need an explanation from me!), and what percentage of people do you think to get to the end of this terminology explanation?

Eating Your Own Dogfood

Apple has several obscure tools for dealing with certificates, including security, certtool, and the Certificate Assistant (and certadmin & Server Admin for Mac OS X Server). Reading their documentation, I keep finding obvious issues, which Apple should have found and fixed already.

I blame this on two main causes. First, each of these tools has a relatively small potential user base. While openssl is available on Macs, Linux, Solaris, and even Windows, certtool is only available for Mac OS X users, and only relevant for command-line users, whereas Certificate Assistant is only relevant for graphical Mac users (personally, I want command-line tools, which I can automate or use in a shell -- likely through an ssh connection). Second, these tools have a very specific primary audience -- it looks like many of them are to support Apple's own internal requirements. If Mac OS X Server needs certificates, someone at Apple writes a tool to build them. If they don't need export, certtool might not get an export option, even if that's core functionality for SSL users in general.

Secondary reasons include ties to proprietary Apple technologies (people who work with certificate files may not want, or be able, to deal with the Apple Keychain), and completeness. The manual page for security, for instance, describes it as incomplete (as certtool is), and some of these are only recently available.

The unfortunate result is that Apple provides a whole mess of almost-baked and almost unknown tools. Even Apple ignores their tools sometimes -- http://developer.apple.com/server/security_ssl.html uses CA.pl, included with OpenSSL, and barely mentions certtool, completely ignoring the Assistant.


The missing private keys are a Keychain Access bug, and fortunately all your certs are created against independent keys by default, which is the right way to do it. Check out the comments for more (and thank you, Perry http://www.cynic.org/perry/).

Separately, I thought I was going crazy, trying to reconcile observed reality with http://developer.apple.com/server/security_ssl.html. The resolution turned out to be simple -- disbelieve! That document apparently describes Panther, and hasn't been updated since it become wrong. It largely discusses how to put a cert into a keychain file for the MOSX mail server, but as of 10.4, Apple's Cyrus installation doesn't use the keychain. I guess updating it for 10.4 would make the body of the article (about certtool and keychains) irrelevant, and nobody's bothered to replace it with something current.

pepper@www:~$ grep tls /etc/imapd.conf
tls_cert_file: /etc/certificates/mail.reppep.com.crt
tls_ca_file: /etc/certificates/ca.reppep.com.crt
tls_key_file: /etc/certificates/mail.reppep.com.key
tls_server_options: use
tls_common_name: mail.reppep.com
tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH

Wednesday, January 17 2007

More Keychain/.Mac Brokenness

.Mac sync doesn't work without saving your password in the Apple Keychain -- BROKEN! I don't want to save my password on a laptop that's likely to get stolen.

If you delete your .Mac password from the keychain, Sync Now from the iSync menu fails with an error, but without an opportunity to enter the password:

Menu: Sync Now Error

In System Preferences:.Mac:Sync, clicking Sync Now generates the same error message with a different icon:

System Preferences: Sync Now Error

After entering a password in System Preferences:.Mac:Sign In, the system pops up a keychain password prompt. If a password is provided, the .Mac password is immediately saved to the default keychain. This is the only keychain access dialog I'm aware of which doesn't offer a checkbox to save the password, and instead forces password saving on -- BROKEN!:

.Mac Keychain Password dialog

No .Mac renewals for me.

Friday, December 29 2006

Woe Is Cyrus

I've said it before, and I'll say it again: Apple should've picked a simpler mail server for Mac OS X Server -- preferably one that used user-accessible mail files or folders, like UW or Dovecot or Courier (I've only used UW, and it wasn't great, but it was simple, and Apple obviously liked plain files over databases enough to customize Blojsom instead of WordPress).

Somebody in Cupertino decided it was a point of pride that MOSX must be able to handle a large-scale enterprise workload, and they picked software that's cluster capable, but who would choose (very nice) Xserves to provide mail service for tens of thousands of users? Apple isn't even actively pursuing this market, although they assure us clustered Xserves running Xsan over Xserve RAIDs would be fast. No way -- in that market, lack of RAID controller redundancy is an instant deal-breaker, just as lack of redundant power was in pre-Intel Xserves.

If you're going to provide an industrial-strength mail server (intended to be no harder to install and configure than a USENET newsfeed -- yowch!) to Mac users, you need to provide more than 2 buttons for troubleshooting Reconstruct (account) and Repair (database). They have never solved a problem for me.

When these fail, Mac users are out in the cold. Apple's documentation is grossly inadequate for troubleshooting, their fora are not helpful, and their build of Cyrus is just different enough to make things more even complicated than they need to be.

I've had database corruption about 3 times in the past, and eventually writhed my way out of it. This time, my $15 CyberPower UPS (which I liked, as it worked better than my $100 APC & Tripp-Lite UPSes) spontaneously expired Christmas Eve while we were at my parents' house. Feh! When we got back home, it wouldn't turn back on.

Apparently this happened at a poor time, and a message got eaten (INBOX #76641, to be exact). Eudora simply couldn't check my INBOX -- the server died every time. Eventually, I got an informative error from SquirrelMail which helped finger the bad message, copied another message to user/pepper/76641., fixed up ownership, and sacrificed a few Tamagotchi while running /usr/bin/cyrus/bin/reconstruct (I'd love to know who decided on /usr/bin/cyrus/bin) with different guessed arguments, until Eudora started sucking down mail as it's supposed to once again.

That was thoroughly unpleasant. Why can't Cyrus survive a missing message? Why can't it warn about a missing message, or an unreadable/root-owned message?

PS-It doesn't help that syslogd periodically stops logging mail traffic. There's probably a better way to make syslogd work again, but rebooting does the trick. Unfortunately, this means I often have no logs of problems since it dies a little later...

Update: Apple just updated Reconstructing cyrus mailboxes in Mac OS X Server 10.3, 10.4, which provides correct usages for the (very picky) reconstruct command, but no explanation whatsoever.

Friday, December 15 2006

iTunes Library Corruption

My 1.5GHz PBG4 has been painfully slow for the last month or so. While I'm game to replace it with a MacBook Pro (Core 2 Duo, baby!), we haven't sold our old apartment yet, so new computers are verboten this month. Reasoning that a) it used to be reasonably snappy, and b) running a 100gb drive with 0-5gb free for a couple years could cause severe fragmentation, I copied almost everything off, repartitioned to get rid of my 10gb OS test partition (which has been very very useful, but is no longer needed for various reasons), and copied stuff back.

Unfortunately, there was a giant fly in my ointment. Since I keep an exact duplicate of my iTunes library on two other machines (via rsync), and the library was more than twice the size of my other OS

Saturday, December 2 2006

Another Stupid Keychain Dependency

For a long time, I ignored the Apple Keychain (see earlier posts for more Keychain travails). I didn't want to keep my passwords anywhere accessible without my intervention. What finally made me give up was the fact that Safari prompted me for my Keychain passphrase on every single flipping page with a text entry field. Eventually I gave up, and started unlocking the keychain. It's very useful, but Apple (Safari) effectively forces many users to use the Keychain by making it so intrusive and unpleasant. Why isn't there a "don't bother me" option in that dialog?

Then when I started using SSHKeychain, the Apple Keychain became much more important to me, because it contained the passphrase for ssh private keys. I am an aggressive locker. When I leave the room, I lock the screen. I do this at home (and irritate Amy), and I do this when I leave the cube farm at work. As a result, I unlock the Mac frequently, with a longer-than-average password. It's a minor nuisance to type one password more than hourly, but if I had to unlock the screensaver, at least one Apple keychain, and one or more ssh private keys, I wouldn't be able to get any actual work done before I bought an Uzi.

With SSHKeychain, I discovered that Apple doesn't support locking Apple keychain(s) when the screensaver locks. Now I know that a major reason for this is that things break when the keychain is locked. In particular, .Mac sync throws all kinds of hissy when it doesn't have access to your .Mac password through a keychain (I've counted 5 different prompts for my Keychain password so different parts of .Mac sync can connect). That's obnoxious, and bad security.

Today's brokenness is related. If you don't have your .Mac password in an unlocked keychain, it's impossible to get a .Mac iChat certificate. Instead you get a bogus error pointing to the "Forgot password" page. I didn't forget my password, you robotic clown! I just won't give it to you for safekeeping (this is on a multi-user server I rarely use, and where I don't want or need saved passwords). I tried entering the password directly into iChat (faster than getting it into the Keychain at that point), but again iChat's Encryption Assisstant failed with a misleading error. As soon as I cached my .Mac password, the Encryption Assistant worked. Two bugs (not accepting manual password entry, and not using a password stored in iChat preferences) + a misleading error message + forcing the user to inferior security (cached password) in order to get a security feature (encrypted chat)!

How perverse is that? Don't answer, please. You'll set me off again.

Tuesday, August 29 2006

EarthLink Sucks

So http://www.brooklynfreespace.org/ needs an Internet connection for 2 700MHz eMacs. Earthlink is cheap for decent speed. I ordered a couple AirPort Extreme cards, a Linksys WRT54GR, and DSL service (and Kid Pix 4 Schools).

Those of you who are eMac savvy will know the first problem already -- although the slot (cleverly concealed behind the CD-ROM door) is about the right size, they need an old (802.11b) AirPort card, not an Extreme card. Fortunately I have my father's old AirPort card, which has outlasted 2 iBooks and still works great (it was worth more than the last iBook I salvaged it from -- original blue, with 3gb drive and dead battery).

Second problem: CD-only drives. Target Disk Mode and my PBG4's DVD drive solved that problem.

Now to the good (bad) stuff:

The WRT54GR comes configured for, with DHCP on. The EarthLink DSL modem (which they charged me $20 for, despite telling me there were no set-up charges), comes configured for with DHCP too. Now it's neat that a $20 piece of hardware is a full router with decent NAT control, but I can't access it through the WRT54GR, since goes to the Linksys, and never reaches the DSL modem.

I got them working with Internet access on Sunday, but cannot reproduce yet. Everything is seriously hampered by a strange error that I keep getting out of the DSL modem when I try to configure it. Suddenly, all the configuration pages start returning this:

Protected Object

This object on the P-660R-ELNK is protected

Then I cannot configure the DSL modem until I reboot the Mac I was using. I suspect it has to do with changing IPs on the Mac (perhaps there's a cookie which it doesn't like seeing come from another IP), but haven't figured it out yet. I sent email to Zyxel, in hopes they'll take pity on me and explain what causes this error (and how to avoid it!).

EarthLink assures me that a full reset of the modem (which they didn't tell me how to do, and isn't covered in their docs) will clear the problem. I doubt it, but we'll see.

So to get support, they want me to use web-based chat. Except that to open a chat session, I have to answer some questions, and click a radio button confirming that the answer wasn't in their knowledge base, and when I Submit the form, I get an error that I didn't check the button. So it's impossible to get support with Safari. I fire up Firefox, and am able to get past that barrier. So much for the first Mac-friendly ISP!

Today, the chat applet didn't drop my characters, or crash Firefox -- a big improvement over 2-3 weeks ago, when I used it to get pre-sales info (and they told me there was no set-up charge).

Also, the school's EarthLink email address was 31 characters. The service is PPPoE, which requires me to enter the address, into a box which only allows entry of 30 characters! I tried without @earthlink, and it didn't work. So I went to EarthLink's support site, and tried to change the email address. I tried three times, and each time it told me I had mis-entered the captcha. It's 5 upper-case letters -- apparently they can't even get that to work in Safari!

So I got someone from EarthLink to change my email address, and I told him that (since they're obviously selling more DSL than dial-up), they should either fix the stupid Zyxel web page to remove the stupid HTML attribute that prevents more than 30 characters in the address field for PPPoE, or warn people when they create email addresses > 30 characters. I asked the rep to tell someone about the 30-character limit, but doubt it happened.

So today (actually yesterday, by now), I sent email explaining the problem to support@earthlink.net. I get back an auto-response telling me to use their web form. I fill out the web form (again, doesn't work in Safari, so I have to use Firefox), and describing my problem, I get a page that tells me they don't take this type of query by email! I try another, and am again stymied.

Oh, well. That much frustrated effort, trying to help EarthLink fix their problems, requires that I point out their idiocy in a public forum.

Update: I managed to send my report via their Feedback form, which warns me not to expect an answer. But perhaps someone will read it.

Not holding my breath.

And again, this was the Mac-friendly ISP! (Open Door Networks was cool, but never local to me) I remember seeing EarthLink folks getting started at MacLeak's MVB events pre-Expo, back in the day!

Update: No thanks to Earthlink, I eventually figured out that if I used a self-assigned IP address (instead of one assigned by the DSL modem's DHCP server), I always got this error. Dumb. People who use static addresses are much more likely than average to use the DSL modem's configuration interface.

Tuesday, August 22 2006

Apple Keychain: Problems with Intrusiveness & Encrypted Disk Images

I find the Apple Keychain very useful, but need to protect my keys -- apparently more than Apple anticipated. On all my machines, I used to have my login keychain set to close after a couple hours of "inactivity". This was very intrusive, as it also locked SSHKeychain, even if I was actively using ssh keys (since Keychain didn't know SSHKeychain was actively being used). I recently discovered the security command. Now I have a cron job which flushes my login keychain when I'm done working for the day/night on all my machines, and it's enabled me to crank up login's inactivity timer. Twice a day, cron runs "security lock-keychain ~/Library/Keychains/login.keychain".

Locking the Keychain has revealed several problems (which I have reported to Apple). First, .Mac syncing assumes the Keychain is always unlocked. This makes a certain amount of sense, but means .Mac syncing doesn't really work unless you leave your .Mac password always available in a keychain. To address this, I use a separate keychain (lowsec) for the relatively low-security .Mac password -- lowsec never locks itself (until I reboot). I'm not happy about this, but it's the best I could do without giving up the one really useful .Mac service.

This leads to the second problem (a straight bug): extra password prompts. If I use my account password for my login keychain, it should be automatically unlocked whenever I unlock the screensaver (just as it's unlocked when I log in after a reboot). But it isn't. I still get prompted to unlock my login keychain after unlocking the screensaver. What's worse, I often get a couple prompts for my login keychain, and on an untrusted laptop I typically get a couple more for lowsec (for a stalled .Mac sync, and to set my iChat status). On the laptop, SSHKeychain is set to lock my keychains when it sleeps, and SSHKeychain is not selective -- it locks all keychains (also reported to Bart).

The untrusted laptop found another serious problem. I don't trust FileVault, so I created a 2gb AES-encrypted disk image with Disk Utility. I moved ~/.ssh and ~/Library/Keychains (yes, I know about ~/Library/Caches/) onto the disk image, and created symlinks from the original locations. The bug I found is that, if the Keychain can't be used (perhaps, say, because ~/Library/Keychains is a dangling symlink, and it's configured to use one or more keychains there), Disk Utility is unable to present the password dialog for an encrypted disk image. This meant because I'd "broken" the Keychain by dismounting the disk image containing my keys, I was unable to remount the disk image to "fix" the keychain, because the broken keychain was preventing disk mounting! I escaped the vicious circle by moving aside the symlink and creating a new ~/Libraries/Keychain directory, which un-broke the Keychain enough that I could get the disk image back online. Now I leave ~/Library/Keychains/lowsec in the normal place, so the Keychain is functional, and manually specified the locations of my (normally unavailable and mostly empty) keychains on the disk image, so they're available when it's mounted.

Friday, August 11 2006

Security Flaws: AFP-over-SSH Broken

After a discussion with Rich Mogull, where we both agreed that AFP is a threat (note that Apple fixed 4 different AFP threats in Security Update 2006-004), I decided to require ssh tunneling for AFP connections to www.reppep.com. Apple provides a neat feature for automatically tunneling AFP through ssh, but unfortunately it's broken in half a dozen ways...

My initial report:

It is impossible to connect to an AFP server without access to port 548 -- this should work if ssh is available, and AFP-over-SSH is enabled. Instead, with 548 blocked by a firewall, the AFP connection times out -- even using an alias created when connected via AFP-over-SSH.

Connect To Server should accept afps://host as a scheme that specifies AFP-over-SSH. Instead it gets converted to afp://afps/host, which is wrong and nonfunctional.

It's impossible to require ssh for AFP from the server.

It's impossible to support AFP on the server without leaving port 548 open, even though with ssh tunnelling 548 shouldn't be needed.

Note: These are not exploits, but they are real problems with the security of Mac OS X (Server

page 2 of 2 -