I’ve started writing something I hope will be an ebook called Take Control of SSH. It’s an amazingly long process, so here’s an excerpt on private and public key authentication, the most important part of SSH. Note that this was not written as a stand-alone piece, so please excuse the rough edges.

If you’d like to read the rest, please vote for Take Control of SSH.

Part II: Public Key Authentication (Practice)

Private keys are much more secure than passwords (which must be short enough for people to remember and easy to type), but fundamentally they’re still large numbers which function as password equivalents. This means that anyone who has your private key can use it to authenticate as you to an SSH server. Files containing private keys (such as ~/.ssh/id_rsa) are therefore as security sensitive as passwords.

On the server side, public keys serve identify users to the SSH server. If someone can manipulate your ~/.ssh/authorized_keys file, they can insert their own key and impersonate you. To protect against this, sshd ignores authorized_keys files with insecure permissions. Unfortunately, sshd doesn’t complain about this in a visible way, so mysterious failures of public key authentication are often traced back to inadequate permissions. The rule is that the ~./ssh/authorized_keys file and all the directories that contain it, up to the root of the filesystem, must not be writable by any account except the owner. File system permissions cannot stop a system administrator from accessing the key files (both private and public, although the private keys should be encrypted), of course. Similarly, anyone with access to the machine or unencrypted backups could copy the key files.

Create a Public/Private Keypair

To get started with public key authentication, first generate a new rsa key in Terminal with “ssh-keygen -t rsa“. Let ssh-keygen create ~/.ssh/id_rsa, and provide a good password you will remember, because keys cannot be recovered if you forget your password. ssh-keygen will create ~/.ssh if it doesn’t already exist, as well as ~/.ssh/id_rsa (your new private key, encrypted with the provided passphrase) and ~/.ssh/id_rsa.pub (your new public key).

pepperbook:~ sample$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/sample/.ssh/id_rsa): 
Created directory '/Users/sample/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/sample/.ssh/id_rsa.
Your public key has been saved in /Users/sample/.ssh/id_rsa.pub.
The key fingerprint is:
ba:d1:0c:43:93:a8:d5:5a:b8:a2:4b:29:ae:34:ee:3f sample@pepperbook.reppep.com
pepperbook:~ sample$ ls -ld .ssh
drwx------   4 sample  sample  136 Apr  6 00:22 .ssh
pepperbook:~ sample$ ls -l .ssh
total 16
-rw-------   1 sample  sample  736 Apr  6 00:22 id_rsa
-rw-r--r--   1 sample  sample  618 Apr  6 00:22 id_rsa.pub
pepperbook:~ sample$ 

To use the new public key for logging into this account on this machine, copy id_rsa.pub to authorized_keys: “cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys“. Test it with “ssh 127.0.0.1“. First, you will be asked to accept the public key for 127.0.0.1 (recall that keys are used to identify servers, as well as users); it’s then automatically copied into ~/.ssh/known_hosts. Second, you will be asked for the passphrase for your private key, instead of the account password for the remote system (note that the passphrase might be the same same as the account password). Finally, you will get a UNIX shell dmg pt on the “remote” system once the ssh session is established. Log out to terminate the ssh session.

Warning: Some Mac OS X versions suffer confusion between the current IPv4 Internet Protocol and the newer IPv6 protocol; IPv6 enables more computers to connect to the Internet but is not yet in wide use. To avoid these issues, use 127.0.0.1 in commands instead of localhost. They should be equivalent, but are not (at least in early versions of Mac OS X 10.4 “Tiger”, and I believe in “Panther” as well). Fortunately, Apple fixed this by 10.4.8, but it caused a lot of confusion before that, and still does for people using older versions.

pepperbook:~ sample$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
pepperbook:~ sample$ ssh 127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is 8a:d3:22:82:f5:2b:88:f0:20:1b:72:bd:aa:30:e0:e2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.
Enter passphrase for key '/Users/sample/.ssh/id_rsa': 
Welcome to Darwin!
pepperbook:~ sample$ ls -l .ssh
total 32
-rw-r--r--   1 sample  sample  618 Apr  6 00:23 authorized_keys
-rw-------   1 sample  sample  736 Apr  6 00:22 id_rsa
-rw-r--r--   1 sample  sample  618 Apr  6 00:22 id_rsa.pub
-rw-r--r--   1 sample  sample  219 Apr  6 00:23 known_hosts
pepperbook:~ sample$ exit
logout
Connection to 127.0.0.1 closed.
pepperbook:~ sample$ 

Now that you’ve confirmed your public and private keys work, it’s time to copy authorized_keys to some other SSH servers. First, make sure there’s a suitable ~/.ssh directory, next send the file, and then login using the key to confirm it works.

pepperbook:~ sample$ ssh www "mkdir -p .ssh ; chmod 700 .ssh"
The authenticity of host 'www (66.92.104.200)' can't be established.
RSA key fingerprint is 53:66:e9:b5:92:e1:5f:d9:71:fa:87:7b:35:99:f2:d3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www,66.92.104.200' (RSA) to the list of known hosts.
Password:
pepperbook:~ sample$ scp ~/.ssh/authorized_keys www:.ssh/
Password:
authorized_keys                               100%  618     0.6KB/s   00:00    
pepperbook:~ sample$ ssh www
Enter passphrase for key '/Users/sample/.ssh/id_rsa': 
Welcome to Darwin!
www:~ sample$ exit
logout
Connection to www closed.
pepperbook:~ sample$ 

Once that’s tested, you’re ready to copy authorized_keys to more computers. You can also put the id_rsa file on other machines, but be careful with this file! Don’t put it on any machines you don’t completely trust.

Remember that you can use multiple private and public keys. The id_rsa and id_rsa.pub filenames are just defaults — keys don’t need to have any special names, although agents tend to assume that the public key filename is the private key filename plus “.pub“, and may get confused otherwise.

SSH Authentication Agents

So far you’ve replaced password authentication with public key authentication. This is valuable, because it means someone who controls an sshd server which you log into will not get a chance to steal your password. The way SSH handles plain password authentication (against the built-in UNIX or Windows account databases) is by taking the password typed in and sending it through the ssh connection; at the other end the username and password are decrypted and checked against the remote account’s password on file. Both the local ssh program you run and the remote sshd program work with the unencrypted password. There have been major incidents in the past where a popular site (such as SourceForge.net) was broken into, and the crackers recorded all the usernames and passwords used to ssh into SourceForge until people noticed something was wrong. The crackers then used those username/password pairs to log into other sites. It was a colossal mess. Public keys avoid this risk, because the remote system only gets the public key. It doesn’t need the private key, and cannot determine it from the public key.

Warning: It is possible to use unencrypted private keys, but this is generally a very bad idea, because anyone who gains access to the private key file (trivial, with physical access to the computer) then controls that identity, and can impersonate or eavesdrop on the owner. About the only time unencrypted private keys are appropriate is for automated tasks, run when nobody is available, if using an agent is not feasible. Such keys should be restricted as much as possible in authorized_keys, and the accounts they access should be restricted as much as possible as well.

Key agents make use of SSH much more efficient, at the cost of a small reduction in security. Agents read encrypted key files; the agent keeps the unencrypted key in memory (never in a file on disk), making it immediately accessible without further decryption or password entry. This makes use SSH dramatically more efficient. Without an agent, connecting to 5 systems 3 times each would require typing one’s password correctly 15 times; with a loaded agent, all the connections are automatic. As an example, consider checking a file on several machines. An SSH agent automatically provides the private key for each connection, making large-scale operations quick and convenient.

The risk in using an agent is that anyone who can control the agent or its socket (such as a root user) can use it to authenticate, although the private keys themselves are not available directly from the agent. This makes agents (and agent forwarding) unsuitable for use on untrusted machines.

SSHKeychain

On the Mac, SSHKeychain is an excellent SSH agent. It can automatically load keys when needed, forgetting them when the system sleeps. SSHKeychain is accessible through the Dock or the SSHKeychain menu, and integrates with Apple’s Keychain, optionally automatically loading and unloading SSH keys as the Apple Keychain unlocks and locks, and it works with other SSH-supporting programs such as BBEdit.

Once you’ve created and tested your private/public keypair, there are several simple steps to activate SSHKeychain.

1. Open SSHKeychain.dmg to mount it.
2. Install SSHKeychain (I just dragged SSHKeychain.app into /Applications).
3. Unmount SSHKeychain.dmg.
4. Open /Applications/SSHKeychain.app.
5. Configure SSHKeychain’s preferences (accessible from the new SSHKeychain menu on the right side of the menu bar); the critical parts are to enable “Manage (and modify) global environment variables”, and configure specify your private keys if you used nonstandard filenames.
6. Use Agent > Add All Keys from SSHKeychain’s menu to confirm it can load your encrypted keys, and that now you can ssh into systems without entering a password. If you check “Add passphrase to the Apple keychain”, SSHKeychain will no longer prompt you for the key’s passphrase, although it may prompt to unlock your keychain to retrieve the key’s passphrase.
7. Set SSHKeychain to run automatically (I control-clicked on its Dock icon and set Open at Login).
8. From now on, whenever you log into Mac OS X, SSH programs will use SSHKeychain for key management.

The SSHKeychain menu shows either of two icons. When it has one or more keys loaded, it shows a keyring with three keys. When it doesn’t hold any keys, the ring is missing, but the keys are still shown.

There are a couple bugs in Mac OS X 10.4.x’s authentication and the Apple Keychain. In theory, unlocking the screensaver should unlock the user’s default keychain (if it uses the account’s login password, which is Apple’s default configuration), but this doesn’t work properly. Additionally, while a keychain is unlocked, the system does not need to prompt the user to “unlock” that keychain again until it’s re-locked, but this is also broken — Mac OS X prompts to “unlock” keychains that are already unlocked.

In reality, if the system is running with the screensaver and Keychain locked, applications tend to bring up their own Keychain dialogs behind it. Unlocking the screensaver or the Keychain does not dismiss these dialogs. The upshot is that, with SSHKeychain set to lock the Apple Keychain on screen activation (its default behavior), unlocking the screensaver may reveal multiple Keychain dialog boxes which must each be addressed individually. With .Mac synching set to Auto, I often saw three authentication dialogs when I unlocked the screensaver.

As a partial workaround, open Keychain Access, create a new keychain (I called mine “lowsec”), select the new keychain, and use “Edit > Change Settings for Keychain “lowsec” to disable locking entirely for this keychain. Then go back to your main (login) keychain, and move the “.Mac password” item (its name will be your .Mac username) from the main login keychain into the new one. Additionally, enable “Use custom security settings” in SSHKeychain, and set “On screensaver:” to “No action”. With this configuration, the lowsec keychain will be unlocked once at login time and then available until logout or reboot. For higher security I set my main login keychain to automatically lock after 30 minutes of inactivity. Ideally, both Keychain Access and SSHKeychain would allow locking specific keychains when the screen locks, leaving other keychains unlocked, but this is not currently implemented.

Key Restrictions and Forced Commands

It can be very useful to provide restricted access to an account. In SSH, this is possible by adding restrictions to specific keys in authorized_keys. Some of the possible restrictions include disabling interactive login; restricting agent, X11, or port forwarding (tunneling); and specifying a “forced command”, which is always run when the associated key is used, regardless of what is requested by the ssh client. These restrictions are particularly useful for non-interactive tasks such as backup scripts. Such tasks may require SSH connectivity with unencrypted private keys, which should not provide unrestricted ssh access.

Note that multiple keys may provide access to the same account. This is handy for people sharing single accounts, or for using special-purpose keys with different forced commands. For more information on key restrictions and forced commands, see the “AUTHORIZED_KEYS FILE FORMAT” section of the sshd manual page.

Public keys can be further restricted by allowing access to carefully circumscribed accounts; unencrypted public keys which give access to “real” user or root accounts should be avoided as much as possible.

Raising the Bar: Requiring Stronger Security

There are several ways to protect a server against ssh-based attacks, including firewalls, TCP Wrapper, not creating or distributing UNIX passwords (forcing ssh public key authentication), and disabling password access for all accounts or the root account. To read about using firewalls and TCP Wrapper with ssh, please vote for Take Control of SSH.

UNIX passwords present several problems for administrators. What legitimate users can remember and type (generally considered to be 8 letters and numbers) is a small enough range of possibilities for attackers to try all possibilities. “Account lockout” is a feature of some systems (including Mac OS X Server) to disable accounts after several failed guesses — which often identifies an attack. Unfortunately, this means legitimate users get blocked when their accounts are attacked, and locking legitimate users out of their own accounts is a successful attack (although not as serious as gaining illicit access). System administrators would often prefer to avoid this by not allowing password access at all. On Mac OS X, it is difficult to set up an account without a password; it’s easier to create a long random password (12+ characters — Keychain Access can do this for you), and never write it down or give it to the account user, requiring public key authentication or some other high security authentication (such as smart cards) instead. On other systems, it’s easy to simply not set UNIX passwords for accounts (although there are complications).

OpenSSH has its own features to help force users to use public keys, blocking password guessing over the Internet. If you have enabled the root account (which is technically accomplished by setting a password for it), you should definitely set “PermitRootLogin without-password” in /etc/sshd_config; this will prevent people from breaking into the root account by guessing its password — an amazingly common and disturbingly successful Internet attack. Even better, set “PasswordAuthentication no” to prevent sshd from accepting passwords for any account — thus requiring keys for ssh access. For more about how to configure sshd and ssh, and enhance security with ssh, please vote for Take Control of SSH.

I’ve started writing something I hope will be an ebook called Take Control of SSH. It’s an amazingly long process, so here’s an excerpt — the coolest trick, which initially made me want to write about SSH. This is for all those of you (us) who support family and friends remotely.

If you’d like to read the rest, please vote for Take Control of SSH.

Reverse Tunnel (Meet in the Middle)

This is a somewhat strange but very powerful feature. One of the problems with modern-day networking is that most computers are protected behind firewalls and/or Network Address Translation (NAT), meaning it may not be possible to connect directly to the computer you want to reach. For example, I sometimes want to remotely control my father’s PowerBook to help with a problem, but he has a private dynamic address, behind an AirPort Extreme base station which itself has another dynamic address and provides NAT. Even if my father told me his IP address and his base station’s address, I wouldn’t be able to reach his PowerBook. Reverse tunneling works around this by allowing him to “project” a tunnel to another computer. If I can reach that computer, the tunnel connects back to his computer. Because he establishes the reverse tunnel from his computer as a client, he doesn’t have trouble with (Network Address Translation) NAT, and he doesn’t even have to know his own IP address. For reverse tunneling to work, my father and I both need access to a computer running sshd. In this example, my PowerBook is called pb, his is called dad, and the server is www.reppep.com.

Note that reverse tunneling provides access to the tunneling computer, and so could be a violation of network security policies. The tunnel is restricted (only accessible from the gateway computer, or other systems with access to it), but it is important to check with the network administrators before implementing a reverse tunnel. In addition, reverse tunneling is fairly complicated. A simpler option is to make SSH directly accessible, but unfortunately this often cannot be arranged. If you’d like to read more about configuring dynamic DNS and firewalls for direct SSH access, please vote for Take Control of SSH.

To use Remote Desktop through a reverse tunnel and an intermediate system, there are several steps (this can be simplified if the client is directly on the Internet and accessible via ssh). There might be as many as 3 separate encryption/decryption cycles in this scenario, but performance is generally decent.

These instructions were originally written for Apple Remote Desktop 2, but apply equally to Remote Desktop 3. Generally, I strongly suggest connecting with Remote Desktop 3’s “Encrypt all network data (more secure)” option, but in this case it doesn’t do quite what is needed, and would only add a redundant extra layer of encryption.

1. On the computer to be controlled (dad in this example), make sure Apple Remote Desktop is enabled in System Preferences:Sharing:Services.
2. On the computer to be controlled (dad), open a couple ports on the server and point them at the ssh and VNC servers on dad, by typing the following into a Terminal window. The sleep command ensures the tunnels will be accessible for at least an hour — they stay open automatically while in use:

   ssh -R 6900:127.0.0.1:5900 -R 6922:127.0.0.1:22 www.reppep.com sleep 3600

3. On the machine running Remote Desktop.app (pb in this example), set up tunnels to those ports on the server:

   ssh -C -L 6900:127.0.0.1:6900 -L 6922:127.0.0.1:6922 -o HostKeyAlias=dad www.reppep.com

4. Now you can log into dad with:

   ssh -p 6922 www.reppep.com

5. Alternatively, you can connect to 127.0.0.1:6900 with Remote Desktop (or port 6900/display 1000 in a VNC client), and control dad.

Note that if the firewall in Mac OS X (non-Server) is on, and any services are enabled in System Preferences:Sharing:Services, the appropriate network ports are automatically opened in the firewall, accessible to the entire Internet; this is overly exposed. OpenSSH can be restricted with TCP Wrapper; the Apple Remote Desktop agent (which is a VNC server at its core) cannot; VineServer has an option to only listen on the loopback address, so external hosts cannot connect — or even detect the presence of a VNC server — without first establishing an ssh tunnel.

http://www.redstonesoftware.com/products/vine/server/

To make this easier for my father, I made sure he uses the same login name on www.reppep.com as on dad, and I gave him a double-clickable file (ssh-tunnel.command) to run step #2 (the part he has to do each time I’m going to help him), in his Dock, to establish the ssh tunnel. Files ending with .command are automatically executed by Terminal when opened in Mac OS X.

To see what TCP ports are accepting connections, use “netstat -an | grep LIST“. Step #2 creates tunnels on the server’s ports 6900 & 6922 (connected back to dad), and step #3 opens the same port numbers on the client (pb), tunneled to the server.

When tunneling ssh connections through ssh, the ssh program tends to show a “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” alert, because ssh gets confused about which ssh server it’s connecting to. The short-term fix is to edit ~/.ssh/known_hosts and remove the entry for 127.0.0.1 or localhost. If you would like to learn more about the cause and other options for dealing with it (including HostKeyAlias, shown in the example above), please vote for Take Control of SSH.

People are also calling it the Pavilion and the Big Top (I like Big House), and we’re now supposed to move in April 27th (previously it was set for March 15th, but I guess we’re not going to make that one). Last week, moving boxes showed up in our current office — that wasn’t encouraging.

We’re discussing steps to alleviate the crunch, like swapping out desktops for laptops and virtual machines, but I’m somewhat surprised there’s no big movement in this direction. Personally, I’m hoping to get rid of 3 desktops and upgrade a laptop. Mac Pro octo (when released) should be a dandy VM host too.

It would be helpful if we could do some telecommuting, but I’m not sure if the Powers That Be will allow it. We’ll have to see how bad the crowding and noise are — the real experience may change everyone’s thinking (or it might not be too bad, but that’s a lot to hope for). The partitions certainly were higher than I expected, and the cubes less tiny, although if we double up, they will be very cramped. On the bright side, I hear the city will only give a 5-year permit for the tent, which is better than I expected. It’s not a good thing when you’re relieved at only spending 5 years in a large semi-permanent “building” with outside bathrooms. But hey, at least we’ll have good connectivity. Oy vey!

Anyway, new pics are up.

Today’s tip is brought to you by the letters SUNW.

As we migrate from encrypted passwords to public keys for ssh access, one of the problems we have is how to make sure an account is disabled across many machines (we’re not up to authenticating against a central LDAP or AD directory yet).

Given the variations in home directory location, eccentricities of escaping commands over ssh connections (dsh makes it even worse), difficulty in recognizing public key strings, and level of paranoia appropriate when closing an account, it’s very difficult to be certain a person’s access has been entirely cut off (especially if they had legitimate access to elevate privilege).

There are simply a lot of files in a lot of different places to check and edit, and no way to know you’ve found them all without spending some quality time with each host, when we want something quick and complete.

On Solaris (but not on RHEL4), if the password field in /etc/shadow is set to *LK*, the account is not accessible via OpenSSH and public keys, although it is via local su. Just to keep things interesting, OpenSSH prompts for a password, even though it knows the account is locked! If the password field is ‘*‘, ‘!‘, or a real encrypted password prefixed with ‘!‘ (a standard way of temporarily disabling a password), ssh access is permitted with a valid public key.

That doesn’t guarantee the user didn’t bury their public key in someone else’s authorized_keys file, but doing this to other individuals would be clearly malicious and actionable, and doing it to the system accounts is more reasonable to check; better is to have a policy of simply replacing their authorized_keys files from known-good copies whenever there’s a change.

Update 2007/05/11: In Mac OS X Server, you can uncheck “access account” in Workgroup Manager, but still log in with a public key.

Under OpenSSH on Solaris, authorized_keys (and the .ssh directory) can be owned by root, handy for centralized account management, but under RHEL4 they must be owned by the user.

I think I’ve seen this before, but thanks to blake for the reminder. If you enter /System/Library/Frameworks/ScreenSaver.framework/Resources/ScreenSaverEngine.app/Contents/MacOS/ScreenSaverEngine -background in a Terminal window, your screensaver will run as a Desktop background. Slide shows of Julia on 2 monitors while I’m able to continue working are nice.

The Canon SD800 IS started taking a long time to recharge the flash, and exhibiting the lag issues I’ve seen before with the S400, so I changed the battery and they went away. I’d gotten up to image 365 (videos, of which I’ve taken about 5, increment the image count). This is very good battery life.

First flaw: the low charge warning doesn’t come soon enough. It should be before the camera’s behavior is affected, with enough margin that people will normally be able to keep using the camera for a while after seeing the warning before it becomes problematic, since most of us don’t want to stop shooting until we can recharge or get a fresh battery.

Second issue: Image Stabilization doesn’t work when the camera is held at a 90° angle. Worse, the image stabilization icon (either a hand with motion bars or a circle with motion bars) doesn’t change when IS is disabled due to orientation. I only know it’s not active because I read the manual.

Overall, I like the camera very much, even though I’m disappointed they didn’t fix the recharge warning (which has been this way since the S400 was new, at least).

I was up on the 8th floor of the Rockefeller Research Building (RRB), and got captured by the prospect looking down, south onto the FDR and the East River, and separately west onto York at 63rd Street (and the Peggy Rockefeller Plaza). I took a bunch of pictures, including a few of the crowded mess that is my cube. Our move to the Super-Tent has been pushed back from March to April (tax day?), so I’ll have to clean up and dump a bunch of stuff soon, to fit into the new cube-farm.

But I was reminded that, overall, RU is a good place to be. It’s why I came back after leaving several years ago, and why I’ve stayed this time (I’m in my 7th year this round).

FYI: Our web group has posted an excellent Interactive Campus Map.

Oh, and the camera is still going strong after having taken a total of 188 pictures & movies combined since its initial charge.

I like the SD800 IS a lot. The face recognition (although it still feels like a computationally insoluble problem to me) actually works, and the image stabilization does too. It enables me to take slower shots (and use flash less often), which is more likely to get a blur effect on running kids — often fun, sometimes unacceptable. I’ve taken 148 photos and a few 640×480x15fps videos and the battery is still fine.

My only complaint about the S400 is that it only had a “battery low” warning, and inevitably once I saw that warning, the camera’s performance was already starting to suffer (particularly flash recharge times). Something showing 0-10%, 10-20% .. 90-100% charge would a major improvement, but I haven’t seen anything like a charge indicator on the new one either, yet. We’ll see if it gives me enough notice. If not, I guess I’ll just have to get a spare battery, but they ain’t cheap.

Almost everything is logical or as expected from the S400. I was surprised that image stabilization doesn’t work when the camera is rotated 90°, though.

The SanDisk 2gb Ultra II SD Plus USB Card is indeed excellent.

I’m writing about SSL/TLS on Mac OS X, so I’ve been playing around with certtool (a very nice text-based interactive tool for working with certificates, based around the Apple Keychain), and security (a more powerful tool for dealing with keychains in general).

I’ve discovered that not only can’t certtool export the certificates (or keys) it generates (it only exports CSRs, which is important but insufficient), but “security export” doesn’t work for me either. I can’t tell if it’s pilot error, but the documentation is sparse and inconsistent.

Needless to say, I left some work in Radar for Apple’s doc folks and security programmers this week.

More confusing, when I generate a certificate with certtool it shows up as expected in Keychain Access, but I don’t see the private key there. My understanding of certificates is that there must be a private key for every certificate (although one key might be used by several certificates), so where is it? Does every keychain have am embedded and invisible private key, which you never see? This has implications for people generating certificates, as they might not be able to disentangle and distribute them in the future, if multiple certificates are tied to the same secrete key. If I could get “security export” to work, I could see if it emits the key along with the cert, but so far no joy.

Just to make things more interesting(!), I have private and public keys in my keychains. They’re not SSHKeychain items, which are identified as such (and really just each hold an encryption passphrase for a normal triple-DES encrypted OpenSSH private key).

So I suspect there is a ‘bedrock’ private key in each of my keychains, (probably automatically generated when the keychain is created), which I can’t see or access directly. But I don’t know! This would make sense, as the keychain items are all encrypted, so perhaps they are encrypted with the same private key the public keys are tied to. But keychain icons can be decrypted and re-encrypted with another key; certificates are permanently tied to their private key.

More bugs: When you open Keychain Access and Show Keychains, you select a keychain in the upper-left corner (”Keychains” list), and only see keys from that keychain. But if you type into the upper-right search box, Keychain Access switches to showing all matches across all keychains, while leaving one ’selected’ in the upper-left — except that selection no longer controls which keychain’s items are displayed, so the meaning of the selection changes (goes away, actually).

But the cool part is this

Launch Keychain Access, and under its Keychain Access menu, select Certificate Assistant. The Certificate Assistant (actually a separate program) has explanations of private/public keys and certificates, and assistants for generating CSRs, self-signed keys, even your own CA — all behind a helpful GUI. For bonus features, you can use it to review existing certificates, or even interrogate an HTTPS server for its cert dynamically and review the cert (similar to visiting the site in Safari and clicking the lock icon), and Apple provided some nice automation, so when you create a CA, it gives you a configuration file you can give to other people. If they open the file, the Certificate Assistant launches on the recipient’s Mac, and walks you through sending a CSR back to the CA Mac. Crazy! It claims to have sent an email request for a certificate signing on my behalf, but I didn’t confirm or see it happen, so I’m a bit confused (again). But it’s very cool, no question.

Terminology

It’s very difficult to be clear when talking about keychains. The Apple Keychain is an API, system feature, and file format (and apparently a framework, distinct from the Keychain Manager, SecurityFoundation.framework, and SecurityInterface.framework).

Each user and installation of Mac OS (including Classic) may have one or more keychains. Keychain Access is Apple’s application for managing both the Apple Keychain (including overarching per-user settings) and individual keychains (both personal and system-wide). The Keychain MenuExtra offers a subset of the features in Keychain Access.

SSHKeychain is a third-party application that manages ssh agents, apparently derived from the Gentoo project’s keychain (a popular command-line tool), but tied into the Apple Keychain, which may contain SSHKeychain items. Confused yet?

Even when I am able to keep it all straight, it’s very difficult to explain this stuff to users who don’t know all the pieces (anyone who already knows all the pieces probably doesn’t need an explanation from me!), and what percentage of people do you think to get to the end of this terminology explanation?

Eating Your Own Dogfood

Apple has several obscure tools for dealing with certificates, including security, certtool, and the Certificate Assistant (and certadmin & Server Admin for Mac OS X Server). Reading their documentation, I keep finding obvious issues, which Apple should have found and fixed already.

I blame this on two main causes. First, each of these tools has a relatively small potential user base. While openssl is available on Macs, Linux, Solaris, and even Windows, certtool is only available for Mac OS X users, and only relevant for command-line users, whereas Certificate Assistant is only relevant for graphical Mac users (personally, I want command-line tools, which I can automate or use in a shell — likely through an ssh connection). Second, these tools have a very specific primary audience — it looks like many of them are to support Apple’s own internal requirements. If Mac OS X Server needs certificates, someone at Apple writes a tool to build them. If they don’t need export, certtool might not get an export option, even if that’s core functionality for SSL users in general.

Secondary reasons include ties to proprietary Apple technologies (people who work with certificate files may not want, or be able, to deal with the Apple Keychain), and completeness. The manual page for security, for instance, describes it as incomplete (as certtool is), and some of these are only recently available.

The unfortunate result is that Apple provides a whole mess of almost-baked and almost unknown tools. Even Apple ignores their tools sometimes — http://developer.apple.com/server/security_ssl.html uses CA.pl, included with OpenSSL, and barely mentions certtool, completely ignoring the Assistant.

Update

The missing private keys are a Keychain Access bug, and fortunately all your certs are created against independent keys by default, which is the right way to do it. Check out the comments for more (and thank you, Perry http://www.cynic.org/perry/).

Separately, I thought I was going crazy, trying to reconcile observed reality with http://developer.apple.com/server/security_ssl.html. The resolution turned out to be simple — disbelieve! That document apparently describes Panther, and hasn’t been updated since it become wrong. It largely discusses how to put a cert into a keychain file for the MOSX mail server, but as of 10.4, Apple’s Cyrus installation doesn’t use the keychain. I guess updating it for 10.4 would make the body of the article (about certtool and keychains) irrelevant, and nobody’s bothered to replace it with something current.

pepper@www:~$ grep tls /etc/imapd.conf
tls_cert_file: /etc/certificates/mail.reppep.com.crt
tls_ca_file: /etc/certificates/ca.reppep.com.crt
tls_key_file: /etc/certificates/mail.reppep.com.key
tls_server_options: use
tls_common_name: mail.reppep.com
tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH

My Canon PowerShot S400 (4MP, CF) is beginning to die, finally providing an excuse to get a new camera (I have no complaints — it has taken several thousand good pictures). The new SD800 IS (7.1MP, SD) is prettier (thanks for the pointer, Adam), slightly thinner (although it’s also slightly wider, so it isn’t really smaller), with a much larger LCD and much better video modes (I rarely carry my Sony DV video camera).

Even the AC adapter is thinner (and wider), with a more interesting retracting plug (not really significant, but aesthetically nicer).

I’m as impressed with my new SanDisk 2gb Ultra II SD Plus USB Card (good idea, Alex). It cost $60 and the flash is really in one half the SD card; the other half is a USB connector and plastic shell to fill out the SD card shape.

I can’t help but compare it to older devices. By comparison, our original Macintosh had 128k of RAM (the new card holds 16,384 times more!) and 400k floppy drives (5,242 times more), and our first hard drive was a 45mb SyQuest cartridge drive (45 times more), and my Apple ][+ had 48k of RAM (1/43,690th as much as the SD card!). At the time, I remember deciding to get the 16k AppleSoft BASIC upgrade card, but not to max out the RAM with 64k, because I knew I wouldn’t need more than 48k (then, not ever — BillG was crazy!).