I'm writing about SSL/TLS on Mac OS X, so I've been playing around with certtool (a very nice text-based interactive tool for working with certificates, based around the Apple Keychain), and security (a more powerful tool for dealing with keychains in general).

I've discovered that not only can't certtool export the certificates (or keys) it generates (it only exports CSRs, which is important but insufficient), but "security export" doesn't work for me either. I can't tell if it's pilot error, but the documentation is sparse and inconsistent.

Needless to say, I left some work in Radar for Apple's doc folks and security programmers this week.

More confusing, when I generate a certificate with certtool it shows up as expected in Keychain Access, but I don't see the private key there. My understanding of certificates is that there must be a private key for every certificate (although one key might be used by several certificates), so where is it? Does every keychain have am embedded and invisible private key, which you never see? This has implications for people generating certificates, as they might not be able to disentangle and distribute them in the future, if multiple certificates are tied to the same secrete key. If I could get "security export" to work, I could see if it emits the key along with the cert, but so far no joy.

Just to make things more interesting(!), I have private and public keys in my keychains. They're not SSHKeychain items, which are identified as such (and really just each hold an encryption passphrase for a normal triple-DES encrypted OpenSSH private key).

So I suspect there is a 'bedrock' private key in each of my keychains, (probably automatically generated when the keychain is created), which I can't see or access directly. But I don't know! This would make sense, as the keychain items are all encrypted, so perhaps they are encrypted with the same private key the public keys are tied to. But keychain icons can be decrypted and re-encrypted with another key; certificates are permanently tied to their private key.

More bugs: When you open Keychain Access and Show Keychains, you select a keychain in the upper-left corner ("Keychains" list), and only see keys from that keychain. But if you type into the upper-right search box, Keychain Access switches to showing all matches across all keychains, while leaving one 'selected' in the upper-left -- except that selection no longer controls which keychain's items are displayed, so the meaning of the selection changes (goes away, actually).

But the cool part is this

Launch Keychain Access, and under its Keychain Access menu, select Certificate Assistant. The Certificate Assistant (actually a separate program) has explanations of private/public keys and certificates, and assistants for generating CSRs, self-signed keys, even your own CA -- all behind a helpful GUI. For bonus features, you can use it to review existing certificates, or even interrogate an HTTPS server for its cert dynamically and review the cert (similar to visiting the site in Safari and clicking the lock icon), and Apple provided some nice automation, so when you create a CA, it gives you a configuration file you can give to other people. If they open the file, the Certificate Assistant launches on the recipient's Mac, and walks you through sending a CSR back to the CA Mac. Crazy! It claims to have sent an email request for a certificate signing on my behalf, but I didn't confirm or see it happen, so I'm a bit confused (again). But it's very cool, no question.


It's very difficult to be clear when talking about keychains. The Apple Keychain is an API, system feature, and file format (and apparently a framework, distinct from the Keychain Manager, SecurityFoundation.framework, and SecurityInterface.framework).

Each user and installation of Mac OS (including Classic) may have one or more keychains. Keychain Access is Apple's application for managing both the Apple Keychain (including overarching per-user settings) and individual keychains (both personal and system-wide). The Keychain MenuExtra offers a subset of the features in Keychain Access.

SSHKeychain is a third-party application that manages ssh agents, apparently derived from the Gentoo project's keychain (a popular command-line tool), but tied into the Apple Keychain, which may contain SSHKeychain items. Confused yet?

Even when I am able to keep it all straight, it's very difficult to explain this stuff to users who don't know all the pieces (anyone who already knows all the pieces probably doesn't need an explanation from me!), and what percentage of people do you think to get to the end of this terminology explanation?

Eating Your Own Dogfood

Apple has several obscure tools for dealing with certificates, including security, certtool, and the Certificate Assistant (and certadmin & Server Admin for Mac OS X Server). Reading their documentation, I keep finding obvious issues, which Apple should have found and fixed already.

I blame this on two main causes. First, each of these tools has a relatively small potential user base. While openssl is available on Macs, Linux, Solaris, and even Windows, certtool is only available for Mac OS X users, and only relevant for command-line users, whereas Certificate Assistant is only relevant for graphical Mac users (personally, I want command-line tools, which I can automate or use in a shell -- likely through an ssh connection). Second, these tools have a very specific primary audience -- it looks like many of them are to support Apple's own internal requirements. If Mac OS X Server needs certificates, someone at Apple writes a tool to build them. If they don't need export, certtool might not get an export option, even if that's core functionality for SSL users in general.

Secondary reasons include ties to proprietary Apple technologies (people who work with certificate files may not want, or be able, to deal with the Apple Keychain), and completeness. The manual page for security, for instance, describes it as incomplete (as certtool is), and some of these are only recently available.

The unfortunate result is that Apple provides a whole mess of almost-baked and almost unknown tools. Even Apple ignores their tools sometimes -- http://developer.apple.com/server/security_ssl.html uses CA.pl, included with OpenSSL, and barely mentions certtool, completely ignoring the Assistant.


The missing private keys are a Keychain Access bug, and fortunately all your certs are created against independent keys by default, which is the right way to do it. Check out the comments for more (and thank you, Perry http://www.cynic.org/perry/).

Separately, I thought I was going crazy, trying to reconcile observed reality with http://developer.apple.com/server/security_ssl.html. The resolution turned out to be simple -- disbelieve! That document apparently describes Panther, and hasn't been updated since it become wrong. It largely discusses how to put a cert into a keychain file for the MOSX mail server, but as of 10.4, Apple's Cyrus installation doesn't use the keychain. I guess updating it for 10.4 would make the body of the article (about certtool and keychains) irrelevant, and nobody's bothered to replace it with something current.

pepper@www:~$ grep tls /etc/imapd.conf
tls_cert_file: /etc/certificates/mail.reppep.com.crt
tls_ca_file: /etc/certificates/ca.reppep.com.crt
tls_key_file: /etc/certificates/mail.reppep.com.key
tls_server_options: use
tls_common_name: mail.reppep.com
tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH