The Apple Keychain is cool, but also strange and problematic
I’m writing about SSL/TLS on Mac OS X, so I’ve been playing around with certtool (a very nice text-based interactive tool for working with certificates, based around the Apple Keychain), and security (a more powerful tool for dealing with keychains in general).
I’ve discovered that not only can’t certtool export the certificates (or keys) it generates (it only exports CSRs, which is important but insufficient), but “security export” doesn’t work for me either. I can’t tell if it’s pilot error, but the documentation is sparse and inconsistent.
Needless to say, I left some work in Radar for Apple’s doc folks and security programmers this week.
More confusing, when I generate a certificate with certtool it shows up as expected in Keychain Access, but I don’t see the private key there. My understanding of certificates is that there must be a private key for every certificate (although one key might be used by several certificates), so where is it? Does every keychain have am embedded and invisible private key, which you never see? This has implications for people generating certificates, as they might not be able to disentangle and distribute them in the future, if multiple certificates are tied to the same secrete key. If I could get “security export” to work, I could see if it emits the key along with the cert, but so far no joy.
Just to make things more interesting(!), I have private and public keys in my keychains. They’re not SSHKeychain items, which are identified as such (and really just each hold an encryption passphrase for a normal triple-DES encrypted OpenSSH private key).
So I suspect there is a ‘bedrock’ private key in each of my keychains, (probably automatically generated when the keychain is created), which I can’t see or access directly. But I don’t know! This would make sense, as the keychain items are all encrypted, so perhaps they are encrypted with the same private key the public keys are tied to. But keychain icons can be decrypted and re-encrypted with another key; certificates are permanently tied to their private key.
More bugs: When you open Keychain Access and Show Keychains, you select a keychain in the upper-left corner (”Keychains” list), and only see keys from that keychain. But if you type into the upper-right search box, Keychain Access switches to showing all matches across all keychains, while leaving one ’selected’ in the upper-left — except that selection no longer controls which keychain’s items are displayed, so the meaning of the selection changes (goes away, actually).
But the cool part is this
Launch Keychain Access, and under its Keychain Access menu, select Certificate Assistant. The Certificate Assistant (actually a separate program) has explanations of private/public keys and certificates, and assistants for generating CSRs, self-signed keys, even your own CA — all behind a helpful GUI. For bonus features, you can use it to review existing certificates, or even interrogate an HTTPS server for its cert dynamically and review the cert (similar to visiting the site in Safari and clicking the lock icon), and Apple provided some nice automation, so when you create a CA, it gives you a configuration file you can give to other people. If they open the file, the Certificate Assistant launches on the recipient’s Mac, and walks you through sending a CSR back to the CA Mac. Crazy! It claims to have sent an email request for a certificate signing on my behalf, but I didn’t confirm or see it happen, so I’m a bit confused (again). But it’s very cool, no question.
Terminology
It’s very difficult to be clear when talking about keychains. The Apple Keychain is an API, system feature, and file format (and apparently a framework, distinct from the Keychain Manager, SecurityFoundation.framework, and SecurityInterface.framework).
Each user and installation of Mac OS (including Classic) may have one or more keychains. Keychain Access is Apple’s application for managing both the Apple Keychain (including overarching per-user settings) and individual keychains (both personal and system-wide). The Keychain MenuExtra offers a subset of the features in Keychain Access.
SSHKeychain is a third-party application that manages ssh agents, apparently derived from the Gentoo project’s keychain (a popular command-line tool), but tied into the Apple Keychain, which may contain SSHKeychain items. Confused yet?
Even when I am able to keep it all straight, it’s very difficult to explain this stuff to users who don’t know all the pieces (anyone who already knows all the pieces probably doesn’t need an explanation from me!), and what percentage of people do you think to get to the end of this terminology explanation?
Eating Your Own Dogfood
Apple has several obscure tools for dealing with certificates, including security, certtool, and the Certificate Assistant (and certadmin & Server Admin for Mac OS X Server). Reading their documentation, I keep finding obvious issues, which Apple should have found and fixed already.
I blame this on two main causes. First, each of these tools has a relatively small potential user base. While openssl is available on Macs, Linux, Solaris, and even Windows, certtool is only available for Mac OS X users, and only relevant for command-line users, whereas Certificate Assistant is only relevant for graphical Mac users (personally, I want command-line tools, which I can automate or use in a shell — likely through an ssh connection). Second, these tools have a very specific primary audience — it looks like many of them are to support Apple’s own internal requirements. If Mac OS X Server needs certificates, someone at Apple writes a tool to build them. If they don’t need export, certtool might not get an export option, even if that’s core functionality for SSL users in general.
Secondary reasons include ties to proprietary Apple technologies (people who work with certificate files may not want, or be able, to deal with the Apple Keychain), and completeness. The manual page for security, for instance, describes it as incomplete (as certtool is), and some of these are only recently available.
The unfortunate result is that Apple provides a whole mess of almost-baked and almost unknown tools. Even Apple ignores their tools sometimes — http://developer.apple.com/server/security_ssl.html uses CA.pl, included with OpenSSL, and barely mentions certtool, completely ignoring the Assistant.
Update
The missing private keys are a Keychain Access bug, and fortunately all your certs are created against independent keys by default, which is the right way to do it. Check out the comments for more (and thank you, Perry http://www.cynic.org/perry/).
Separately, I thought I was going crazy, trying to reconcile observed reality with http://developer.apple.com/server/security_ssl.html. The resolution turned out to be simple — disbelieve! That document apparently describes Panther, and hasn’t been updated since it become wrong. It largely discusses how to put a cert into a keychain file for the MOSX mail server, but as of 10.4, Apple’s Cyrus installation doesn’t use the keychain. I guess updating it for 10.4 would make the body of the article (about certtool and keychains) irrelevant, and nobody’s bothered to replace it with something current.
pepper@www:~$ grep tls /etc/imapd.conf
tls_cert_file: /etc/certificates/mail.reppep.com.crt
tls_ca_file: /etc/certificates/ca.reppep.com.crt
tls_key_file: /etc/certificates/mail.reppep.com.key
tls_server_options: use
tls_common_name: mail.reppep.com
tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH
Perry The Cynic said,
March 4, 2007 at 1:37 pm
Oh my. So much stuff to comment on… There’s no secret private key hidden in your keychains. There’s only an annoying bug where Keychain Access sometimes doesn’t show newly created keychain items until you quit and restart it. (Something about change notifications going awry.) “certtool c” most certainly makes a private key in one of your keychains.
“Security export†does work - you probably want “-t identities -f pkcs12â€. The security(1) command lacks the ability to neatly specify exactly what to include; the export function in Keychain Access lets you select (say) the private key and certificate and export just that. (Under the hood, they do the same thing.)
The division of labor between security(1), certtool(1), and the Certificate Assistant is - how do I put this - ad-hoc, in a “it just happened like that” kind of way. Certtool was the first. Security(1) came later, and was meant to eventually collect all the shell-level functionality of the security layers. Well, “eventually” hasn’t happened yet, and it’s not in sight. :-) The Certificate Assistant was born out of users’ frustration about the incredibly arcane hoops you have to jump through to manage certificate acquisition. And yes, from an “integrated user experience” point of view, it’s a mess.
When people say “the keychain,” they could mean almost any aspect of the security stack built into Mac OS X. The Security people at Apple refer to “the Sec* layer” for the APIs (because all the function names start with “Sec”). “Keychain” is reserved to an actual storage container for security-related things. Even so, “keychain” can refer to either a particular file format for a cryptographic storage container, or to any storage container supported by the Sec* API layers (which includes keychain files as well as smart cards, LDAP servers, and other stuff). I’d recommend saying “keychain file” for the particular files, and “keychain object” for the API objects (SecKeychainRefs). In a modern Mac OS X system, every user has at least one keychain (~/Library/Keychains/login.keychain). The very act of logging in will make one if you don’t have one. You can make more yourself, of course. The system itself also has a keychain to itself (/Library/Keychains/system.keychain). “Classic†(Mac OS 9) keychain were completely different animals; there’s no data compatibility at all, though the old Mac OS 9 APIs (KC*) are still supported, sort of.
One of the causes (if not excuses) for the terminology vagueness is that Apple has traditionally tried to hide most of this stuff from non-expert users. For a long time, users weren’t to be told that they had a keychain at all. Users are still supposed to be blissfully ignorant of the fact that there are (gasp!) “certificates” used in implementing their security. As a result, a lot of the cooler features got into the system as submarines - they’re there mostly because they were inconspicuous. It’s hard to do comprehensive documentation reviews on a bunch of semi-secret features. On top of all that, a lot of the Open Source code included in Mac OS X resolutely refuses to use anything that is system specific - like keychains, say. So stuff like Apache sticks with openssl because that’s cross-platform. Can’t say that I blame them… much.
If you are interested in these things, you may want to subscribe to http://lists.apple.com/mailman/listinfo/apple-cdsa. Apple’s Security developers hang out there, and have been known to answer good-faith questions about pretty much anything in Mac OS X’s security stack. The list archives should be interesting, too; they go back quite a while now.
Cheers — perry