Extra Pepperoni

To content | To menu | To search

ssh

http://www.openssh.com/

Entries feed - Comments feed

Wednesday, July 27 2011

OpenSSH is smart about cluster hostkeys

Normally, the first time you ssh to a new server, OpenSSH asks for permission to store the server's hostname (and IP) along with its unique ssh hostkey in ~/.ssh/known_hosts. Then if the hostkey ever changes, either because the machine was rebuilt or because you're connected to a different machine (as would be the case if someone intercepted your connection, for instance...), OpenSSH complains loudly that something is hinky:

pepper@teriyaki:~$ ssh cluster uname -a
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The DSA host key for cluster has changed,
and the key for the corresponding IP address 10.0.10.124
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the DSA host key has just been changed.
The fingerprint for the DSA key sent by the remote host is
f7:b0:d4:11:2c:6c:ec:be:96:f0:88:71:d9:26:20:0c.
Please contact your system administrator.
Add correct host key in /Users/pepper/.ssh/known_hosts to get rid of this message.
Offending key in /Users/pepper/.ssh/known_hosts:81
DSA host key for cluster has changed and you have requested strict checking.
Host key verification failed.

This is a nuisance with high-availability (HA) clusters, where multiple nodes may share a single hostname and IP. The first time you connect to a shared IP everything works and you store the hostkey for whichever node accepted your connection. Then it may continue to work for a long time, if you keep connecting to the same node. But when you get a different node at that IP, OpenSSH detects it's a different machine (hostkey), and either the connection fails (if it's non-interactive) or you get the scary warning (if it's interactive). To avoid this, the convention is to ssh directly into individual nodes for administration.

But some of our sequencers use rsync-over-ssh to export data to our Isilon storage clusters, so we had a problem. If we configured them to connect to the VIP (like NFS clients), things would break when they connected to different nodes. But if we configured them to connect to individual nodes, we'd lose failover -- if any Isilon node went down, all of 'its' clients would stop transferring data until it came back up.

I briefly considered synchronizing the ssh hostkeys between nodes, to avoid the hostkey errors, but this is poor security -- if each node has the same hostkey, it's easy for any node to eavesdrop on connections to all its peers with the same hostkey, and changing keys is disruptive.

Fortunately the OpenSSH developers are way ahead of me. If the hostkey is already on file as valid for a known host -- even if there are other conflicting keys on file for the same host -- OpenSSH accepts it.

To set this up, just ssh to each node, then append the cluster hostname and IPs to their entries in ~/.ssh/known_hosts or /etc/ssh/ssh_known_hosts.

cluster-1,10.0.10.101,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-2,10.0.10.102,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-3,10.0.10.103,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-4,10.0.10.104,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...

Monday, November 2 2009

Conditional ssh Configuration: iptables & sshd

I have an Internet server which accepts ssh connections from the Internet. I want to lock down Internet connections more tightly, while leaving connections from the local network a bit more open.

On Mac OS X, the way to do this is through the Match directive in /etc/sshd_config. Unfortunately, RHEL/CentOS 5 & SLES 10 include openssh v4.x, which lacks the Match directive.

Things are complicated by multiple overlapping directives to control password authentication in OpenSSH, which behave differently on different platforms. But the OpenSSH developers have suggested global settings in /etc/sshd_config like:

Protocol 2
ChallengeResponseAuthentication no
PasswordAuthentication no
KbdInteractiveAuthentication no

And overrides along the lines of:

Match 10.0.0.0/8
  PasswordAuthentication yes
  KbdInteractiveAuthentication yes

On CentOS Match isn't available, so we run two different sshd daemons. Here's the init script that runs them both (I couldn't just copy and tweak one, because they'd interfere with each other):

[root@c54 ~]# diff -u /etc/init.d/sshd.c54 /etc/init.d/sshd
--- /etc/init.d/sshd.c54    2009-10-26 19:47:06.000000000 -0400
+++ /etc/init.d/sshd    2009-11-02 13:02:24.000000000 -0500
@@ -96,6 +96,12 @@
        failure $"Configuration file or keys are invalid"
        echo
    fi
+   $SSHD -t -f /etc/ssh/sshd_config.222
+   RETVAL=$?
+   if [ ! "$RETVAL" = 0 ]; then
+       failure $"External configuration file or keys are invalid"
+       echo
+   fi
 }

 start()
@@ -108,7 +114,7 @@
    cp -af /etc/localtime /var/empty/sshd/etc

    echo -n $"Starting $prog: "
-   $SSHD $OPTIONS && success || failure
+   $SSHD $OPTIONS && PID_FILE=$PID_FILE_EXT $SSHD -f /etc/ssh/sshd_config.222 $OPTIONS && success || failure
    RETVAL=$?
    [ "$RETVAL" = 0 ] && touch /var/lock/subsys/sshd
    echo
@@ -117,11 +123,7 @@
 stop()
 {
    echo -n $"Stopping $prog: "
-   if [ -n "`pidfileofproc $SSHD`" ] ; then
-       killproc $SSHD
-   else
-       failure $"Stopping $prog"
-   fi
+   killall $SSHD
    RETVAL=$?
    # if we are in halt or reboot runlevel kill all running sessions
    # so the TCP connections are closed cleanly

Their configurations are slightly different:

[root@c54 ~]# diff -u /etc/ssh/sshd_config /etc/ssh/sshd_config.222 
--- /etc/ssh/sshd_config    2009-11-02 16:59:40.000000000 -0500
+++ /etc/ssh/sshd_config.222    2009-11-02 16:59:46.000000000 -0500
@@ -57,7 +57,7 @@
 # To disable tunneled clear text passwords, change to no here!
 #PasswordAuthentication yes
 #PermitEmptyPasswords no
-#PasswordAuthentication yes
+PasswordAuthentication yes

 # Change to no to disable s/key passwords
 #ChallengeResponseAuthentication yes
@@ -119,5 +119,5 @@
 Subsystem  sftp    /usr/libexec/openssh/sftp-server


-# sshd for external (Internet) connections: no passwords allowed
-PasswordAuthentication no
+# sshd for internal (campus) connections: passwords allowed
+Port 222

The intelligent switching occurs in /etc/sysconfig/iptables, which redirects internal connections to the alternat port (and thus daemon). Unfortunately, due to the way iptables works, 'trusted' connections must be both REDIRECTed and ACCEPTed:

*nat
:PREROUTING ACCEPT
:POSTROUTING ACCEPT
:OUTPUT ACCEPT

-A  PREROUTING -p tcp --dport    22 -s 10.0.0.0/24    -j REDIRECT --to-port   222
# Connections from 10.0.0. are more trusted, so send them to the private sshd on port 222.
# Each subnet must also be ACCEPTed below.

COMMIT


*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT

-A INPUT -j ACCEPT -i lo
-A INPUT -j DROP                       -s 127.0.0.0/8
-A INPUT -j DROP                       -d 127.0.0.0/8
# Allow loopback connections, but drop forged pseudo-local packets

-A INPUT -j ACCEPT -m state --state RELATED,ESTABLISHED
# Continue established connections


-A INPUT -j ACCEPT -p tcp --dport    22
# sshd on port  22, for Internet connections (no passwords)

-A INPUT -j ACCEPT -p tcp --dport   222 -s 10.0.0.0/24
# sshd on port 222, for trusted connections
# Each subnet must also be REDIRECTed above.

COMMIT

Tuesday, April 3 2007

Today's Accomplishment: SSO ssh Upgrades

Today I upgraded our 8 core authentication ("Single Sign On") servers from commercial ssh to OpenSSH (it was a 2-year battle to standardize on OpenSSH, but in the end the right product won).

http://www.openssh.org/

The tricky things are: a) These machines are critical enough that they're not directly accessible from campus, so everything must be done through an intermediary machine. This complicates everything. b) Upgrading ssh is problematic because it's the remote control tool, so working on sshd implicitly interferes with your own control. Fortunately we have good terminal servers, which make this much less problematical; when sshd is down, you can get in the back door to bring it up.

One of the neat things about UNIX is that you can delete a file, but if it's open the file is not actually deleted until that filehandle is no longer in use (when the last filehandle is closed, the disk space is reclaimed), so for non-terminal-server systems, I've actually done the whole upgrade through an sshd binary which is deleted at the beginning of the upgrade, replaced during the upgrade, and still in use until the very end.

I (we) have been doing this long enough, and refined the procedure sufficiently, that the whole upgrade took under 90 minutes total for 8 machines, although there was a lot of prep and follow-up, cleaning up accounts, /etc/sudoers, installing public keys, etc.

Overwriting /etc/passwd, /etc/shadow, and /etc/groups with copies from the intermediate machine was particularly stressful; I was highly relieved that nothing broke.

Saturday, March 24 2007

Take Control of SSH, Draft Excerpt: Public Key Authentication

I've started writing something I hope will be an ebook called Take Control of SSH. It's an amazingly long process, so here's an excerpt on private and public key authentication, the most important part of SSH. Note that this was not written as a stand-alone piece, so please excuse the rough edges.

If you'd like to read the rest, please vote for Take Control of SSH.

Part II: Public Key Authentication (Practice)

Private keys are much more secure than passwords (which must be short enough for people to remember and easy to type), but fundamentally they're still large numbers which function as password equivalents. This means that anyone who has your private key can use it to authenticate as you to an SSH server. Files containing private keys (such as ~/.ssh/id_rsa) are therefore as security sensitive as passwords.

On the server side, public keys serve identify users to the SSH server. If someone can manipulate your ~/.ssh/authorized_keys file, they can insert their own key and impersonate you. To protect against this, sshd ignores authorized_keys files with insecure permissions. Unfortunately, sshd doesn't complain about this in a visible way, so mysterious failures of public key authentication are often traced back to inadequate permissions. The rule is that the ~./ssh/authorized_keys file and all the directories that contain it, up to the root of the filesystem, must not be writable by any account except the owner. File system permissions cannot stop a system administrator from accessing the key files (both private and public, although the private keys should be encrypted), of course. Similarly, anyone with access to the machine or unencrypted backups could copy the key files.

Create a Public/Private Keypair

To get started with public key authentication, first generate a new rsa key in Terminal with "ssh-keygen -t rsa". Let ssh-keygen create ~/.ssh/id_rsa, and provide a good password you will remember, because keys cannot be recovered if you forget your password. ssh-keygen will create ~/.ssh if it doesn't already exist, as well as ~/.ssh/id_rsa (your new private key, encrypted with the provided passphrase) and ~/.ssh/id_rsa.pub (your new public key).

pepperbook:~ sample$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/sample/.ssh/id_rsa): 
Created directory '/Users/sample/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/sample/.ssh/id_rsa.
Your public key has been saved in /Users/sample/.ssh/id_rsa.pub.
The key fingerprint is:
ba:d1:0c:43:93:a8:d5:5a:b8:a2:4b:29:ae:34:ee:3f sample@pepperbook.reppep.com
pepperbook:~ sample$ ls -ld .ssh
drwx------   4 sample  sample  136 Apr  6 00:22 .ssh
pepperbook:~ sample$ ls -l .ssh
total 16
-rw-------   1 sample  sample  736 Apr  6 00:22 id_rsa
-rw-r--r--   1 sample  sample  618 Apr  6 00:22 id_rsa.pub
pepperbook:~ sample$ 

To use the new public key for logging into this account on this machine, copy id_rsa.pub to authorized_keys: "cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys". Test it with "ssh 127.0.0.1". First, you will be asked to accept the public key for 127.0.0.1 (recall that keys are used to identify servers, as well as users); it's then automatically copied into ~/.ssh/known_hosts. Second, you will be asked for the passphrase for your private key, instead of the account password for the remote system (note that the passphrase might be the same same as the account password). Finally, you will get a UNIX shell dmg pt on the "remote" system once the ssh session is established. Log out to terminate the ssh session.

Warning: Some Mac OS X versions suffer confusion between the current IPv4 Internet Protocol and the newer IPv6 protocol; IPv6 enables more computers to connect to the Internet but is not yet in wide use. To avoid these issues, use 127.0.0.1 in commands instead of localhost. They should be equivalent, but are not (at least in early versions of Mac OS X 10.4 "Tiger", and I believe in "Panther" as well). Fortunately, Apple fixed this by 10.4.8, but it caused a lot of confusion before that, and still does for people using older versions.

pepperbook:~ sample$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
pepperbook:~ sample$ ssh 127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is 8a:d3:22:82:f5:2b:88:f0:20:1b:72:bd:aa:30:e0:e2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.
Enter passphrase for key '/Users/sample/.ssh/id_rsa': 
Welcome to Darwin!
pepperbook:~ sample$ ls -l .ssh
total 32
-rw-r--r--   1 sample  sample  618 Apr  6 00:23 authorized_keys
-rw-------   1 sample  sample  736 Apr  6 00:22 id_rsa
-rw-r--r--   1 sample  sample  618 Apr  6 00:22 id_rsa.pub
-rw-r--r--   1 sample  sample  219 Apr  6 00:23 known_hosts
pepperbook:~ sample$ exit
logout
Connection to 127.0.0.1 closed.
pepperbook:~ sample$ 

Now that you've confirmed your public and private keys work, it's time to copy authorized_keys to some other SSH servers. First, make sure there's a suitable ~/.ssh directory, next send the file, and then login using the key to confirm it works.

pepperbook:~ sample$ ssh www "mkdir -p .ssh ; chmod 700 .ssh"
The authenticity of host 'www (66.92.104.200)' can't be established.
RSA key fingerprint is 53:66:e9:b5:92:e1:5f:d9:71:fa:87:7b:35:99:f2:d3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www,66.92.104.200' (RSA) to the list of known hosts.
Password:
pepperbook:~ sample$ scp ~/.ssh/authorized_keys www:.ssh/
Password:
authorized_keys                               100%  618     0.6KB/s   00:00    
pepperbook:~ sample$ ssh www
Enter passphrase for key '/Users/sample/.ssh/id_rsa': 
Welcome to Darwin!
www:~ sample$ exit
logout
Connection to www closed.
pepperbook:~ sample$ 

Once that's tested, you're ready to copy authorized_keys to more computers. You can also put the id_rsa file on other machines, but be careful with this file! Don't put it on any machines you don't completely trust.

Remember that you can use multiple private and public keys. The id_rsa and id_rsa.pub filenames are just defaults -- keys don't need to have any special names, although agents tend to assume that the public key filename is the private key filename plus ".pub", and may get confused otherwise.

SSH Authentication Agents

So far you've replaced password authentication with public key authentication. This is valuable, because it means someone who controls an sshd server which you log into will not get a chance to steal your password. The way SSH handles plain password authentication (against the built-in UNIX or Windows account databases) is by taking the password typed in and sending it through the ssh connection; at the other end the username and password are decrypted and checked against the remote account's password on file. Both the local ssh program you run and the remote sshd program work with the unencrypted password. There have been major incidents in the past where a popular site (such as SourceForge.net) was broken into, and the crackers recorded all the usernames and passwords used to ssh into SourceForge until people noticed something was wrong. The crackers then used those username/password pairs to log into other sites. It was a colossal mess. Public keys avoid this risk, because the remote system only gets the public key. It doesn't need the private key, and cannot determine it from the public key.

Warning: It is possible to use unencrypted private keys, but this is generally a very bad idea, because anyone who gains access to the private key file (trivial, with physical access to the computer) then controls that identity, and can impersonate or eavesdrop on the owner. About the only time unencrypted private keys are appropriate is for automated tasks, run when nobody is available, if using an agent is not feasible. Such keys should be restricted as much as possible in authorized_keys, and the accounts they access should be restricted as much as possible as well.

Key agents make use of SSH much more efficient, at the cost of a small reduction in security. Agents read encrypted key files; the agent keeps the unencrypted key in memory (never in a file on disk), making it immediately accessible without further decryption or password entry. This makes use SSH dramatically more efficient. Without an agent, connecting to 5 systems 3 times each would require typing one's password correctly 15 times; with a loaded agent, all the connections are automatic. As an example, consider checking a file on several machines. An SSH agent automatically provides the private key for each connection, making large-scale operations quick and convenient.

The risk in using an agent is that anyone who can control the agent or its socket (such as a root user) can use it to authenticate, although the private keys themselves are not available directly from the agent. This makes agents (and agent forwarding) unsuitable for use on untrusted machines.

SSHKeychain

On the Mac, SSHKeychain is an excellent SSH agent. It can automatically load keys when needed, forgetting them when the system sleeps. SSHKeychain is accessible through the Dock or the SSHKeychain menu, and integrates with Apple's Keychain, optionally automatically loading and unloading SSH keys as the Apple Keychain unlocks and locks, and it works with other SSH-supporting programs such as BBEdit.

Once you've created and tested your private/public keypair, there are several simple steps to activate SSHKeychain.

  1. Open SSHKeychain.dmg to mount it.
  2. Install SSHKeychain (I just dragged SSHKeychain.app into /Applications).
  3. Unmount SSHKeychain.dmg.
  4. Open /Applications/SSHKeychain.app.
  5. Configure SSHKeychain's preferences (accessible from the new SSHKeychain menu on the right side of the menu bar); the critical parts are to enable "Manage (and modify) global environment variables", and configure specify your private keys if you used nonstandard filenames.
  6. Use Agent > Add All Keys from SSHKeychain's menu to confirm it can load your encrypted keys, and that now you can ssh into systems without entering a password. If you check "Add passphrase to the Apple keychain", SSHKeychain will no longer prompt you for the key's passphrase, although it may prompt to unlock your keychain to retrieve the key's passphrase.
  7. Set SSHKeychain to run automatically (I control-clicked on its Dock icon and set Open at Login).
  8. From now on, whenever you log into Mac OS X, SSH programs will use SSHKeychain for key management.

The SSHKeychain menu shows either of two icons. When it has one or more keys loaded, it shows a keyring with three keys. When it doesn't hold any keys, the ring is missing, but the keys are still shown.

There are a couple bugs in Mac OS X 10.4.x's authentication and the Apple Keychain. In theory, unlocking the screensaver should unlock the user's default keychain (if it uses the account's login password, which is Apple's default configuration), but this doesn't work properly. Additionally, while a keychain is unlocked, the system does not need to prompt the user to "unlock" that keychain again until it's re-locked, but this is also broken -- Mac OS X prompts to "unlock" keychains that are already unlocked.

In reality, if the system is running with the screensaver and Keychain locked, applications tend to bring up their own Keychain dialogs behind it. Unlocking the screensaver or the Keychain does not dismiss these dialogs. The upshot is that, with SSHKeychain set to lock the Apple Keychain on screen activation (its default behavior), unlocking the screensaver may reveal multiple Keychain dialog boxes which must each be addressed individually. With .Mac synching set to Auto, I often saw three authentication dialogs when I unlocked the screensaver.

As a partial workaround, open Keychain Access, create a new keychain (I called mine "lowsec"), select the new keychain, and use "Edit > Change Settings for Keychain "lowsec" to disable locking entirely for this keychain. Then go back to your main (login) keychain, and move the ".Mac password" item (its name will be your .Mac username) from the main login keychain into the new one. Additionally, enable "Use custom security settings" in SSHKeychain, and set "On screensaver:" to "No action". With this configuration, the lowsec keychain will be unlocked once at login time and then available until logout or reboot. For higher security I set my main login keychain to automatically lock after 30 minutes of inactivity. Ideally, both Keychain Access and SSHKeychain would allow locking specific keychains when the screen locks, leaving other keychains unlocked, but this is not currently implemented.

Key Restrictions and Forced Commands

It can be very useful to provide restricted access to an account. In SSH, this is possible by adding restrictions to specific keys in authorized_keys. Some of the possible restrictions include disabling interactive login; restricting agent, X11, or port forwarding (tunneling); and specifying a "forced command", which is always run when the associated key is used, regardless of what is requested by the ssh client. These restrictions are particularly useful for non-interactive tasks such as backup scripts. Such tasks may require SSH connectivity with unencrypted private keys, which should not provide unrestricted ssh access.

Note that multiple keys may provide access to the same account. This is handy for people sharing single accounts, or for using special-purpose keys with different forced commands. For more information on key restrictions and forced commands, see the "AUTHORIZED_KEYS FILE FORMAT" section of the sshd manual page.

Public keys can be further restricted by allowing access to carefully circumscribed accounts; unencrypted public keys which give access to "real" user or root accounts should be avoided as much as possible.

Raising the Bar: Requiring Stronger Security

There are several ways to protect a server against ssh-based attacks, including firewalls, TCP Wrapper, not creating or distributing UNIX passwords (forcing ssh public key authentication), and disabling password access for all accounts or the root account. To read about using firewalls and TCP Wrapper with ssh, please vote for Take Control of SSH.

UNIX passwords present several problems for administrators. What legitimate users can remember and type (generally considered to be 8 letters and numbers) is a small enough range of possibilities for attackers to try all possibilities. "Account lockout" is a feature of some systems (including Mac OS X Server) to disable accounts after several failed guesses -- which often identifies an attack. Unfortunately, this means legitimate users get blocked when their accounts are attacked, and locking legitimate users out of their own accounts is a successful attack (although not as serious as gaining illicit access). System administrators would often prefer to avoid this by not allowing password access at all. On Mac OS X, it is difficult to set up an account without a password; it's easier to create a long random password (12+ characters -- Keychain Access can do this for you), and never write it down or give it to the account user, requiring public key authentication or some other high security authentication (such as smart cards) instead. On other systems, it's easy to simply not set UNIX passwords for accounts (although there are complications).

OpenSSH has its own features to help force users to use public keys, blocking password guessing over the Internet. If you have enabled the root account (which is technically accomplished by setting a password for it), you should definitely set "PermitRootLogin without-password" in /etc/sshd_config; this will prevent people from breaking into the root account by guessing its password -- an amazingly common and disturbingly successful Internet attack. Even better, set "PasswordAuthentication no" to prevent sshd from accepting passwords for any account -- thus requiring keys for ssh access. For more about how to configure sshd and ssh, and enhance security with ssh, please vote for Take Control of SSH.

Take Control of SSH, Draft Excerpt: Reverse Tunnels

I've started writing something I hope will be an ebook called Take Control of SSH. It's an amazingly long process, so here's an excerpt -- the coolest trick, which initially made me want to write about SSH. This is for all those of you (us) who support family and friends remotely.

If you'd like to read the rest, please vote for Take Control of SSH.

Reverse Tunnel (Meet in the Middle)

This is a somewhat strange but very powerful feature. One of the problems with modern-day networking is that most computers are protected behind firewalls and/or Network Address Translation (NAT), meaning it may not be possible to connect directly to the computer you want to reach. For example, I sometimes want to remotely control my father's PowerBook to help with a problem, but he has a private dynamic address, behind an AirPort Extreme base station which itself has another dynamic address and provides NAT. Even if my father told me his IP address and his base station's address, I wouldn't be able to reach his PowerBook. Reverse tunneling works around this by allowing him to "project" a tunnel to another computer. If I can reach that computer, the tunnel connects back to his computer. Because he establishes the reverse tunnel from his computer as a client, he doesn't have trouble with (Network Address Translation) NAT, and he doesn't even have to know his own IP address. For reverse tunneling to work, my father and I both need access to a computer running sshd. In this example, my PowerBook is called pb, his is called dad, and the server is www.reppep.com.

Note that reverse tunneling provides access to the tunneling computer, and so could be a violation of network security policies. The tunnel is restricted (only accessible from the gateway computer, or other systems with access to it), but it is important to check with the network administrators before implementing a reverse tunnel. In addition, reverse tunneling is fairly complicated. A simpler option is to make SSH directly accessible, but unfortunately this often cannot be arranged. If you'd like to read more about configuring dynamic DNS and firewalls for direct SSH access, please vote for Take Control of SSH.

To use Remote Desktop through a reverse tunnel and an intermediate system, there are several steps (this can be simplified if the client is directly on the Internet and accessible via ssh). There might be as many as 3 separate encryption/decryption cycles in this scenario, but performance is generally decent.

These instructions were originally written for Apple Remote Desktop 2, but apply equally to Remote Desktop 3. Generally, I strongly suggest connecting with Remote Desktop 3's "Encrypt all network data (more secure)" option, but in this case it doesn't do quite what is needed, and would only add a redundant extra layer of encryption.

  1. On the computer to be controlled (dad in this example), make sure Apple Remote Desktop is enabled in System Preferences:Sharing:Services.
  2. On the computer to be controlled (dad), open a couple ports on the server and point them at the ssh and VNC servers on dad, by typing the following into a Terminal window. The sleep command ensures the tunnels will be accessible for at least an hour -- they stay open automatically while in use:
    ssh -R 6900:127.0.0.1:5900 -R 6922:127.0.0.1:22 www.reppep.com sleep 3600
  3. On the machine running Remote Desktop.app (pb in this example), set up tunnels to those ports on the server:
    ssh -C -L 6900:127.0.0.1:6900 -L 6922:127.0.0.1:6922 -o HostKeyAlias=dad www.reppep.com
  4. Now you can log into dad with:
    ssh -p 6922 www.reppep.com
  5. Alternatively, you can connect to 127.0.0.1:6900 with Remote Desktop (or port 6900/display 1000 in a VNC client), and control dad.

Note that if the firewall in Mac OS X (non-Server) is on, and any services are enabled in System Preferences:Sharing:Services, the appropriate network ports are automatically opened in the firewall, accessible to the entire Internet; this is overly exposed. OpenSSH can be restricted with TCP Wrapper; the Apple Remote Desktop agent (which is a VNC server at its core) cannot; VineServer has an option to only listen on the loopback address, so external hosts cannot connect -- or even detect the presence of a VNC server -- without first establishing an ssh tunnel.

http://www.redstonesoftware.com/products/vine/server/

To make this easier for my father, I made sure he uses the same login name on www.reppep.com as on dad, and I gave him a double-clickable file (ssh-tunnel.command) to run step #2 (the part he has to do each time I'm going to help him), in his Dock, to establish the ssh tunnel. Files ending with .command are automatically executed by Terminal when opened in Mac OS X.

To see what TCP ports are accepting connections, use "netstat -an | grep LIST". Step #2 creates tunnels on the server's ports 6900 & 6922 (connected back to dad), and step #3 opens the same port numbers on the client (pb), tunneled to the server.

When tunneling ssh connections through ssh, the ssh program tends to show a "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" alert, because ssh gets confused about which ssh server it's connecting to. The short-term fix is to edit ~/.ssh/known_hosts and remove the entry for 127.0.0.1 or localhost. If you would like to learn more about the cause and other options for dealing with it (including HostKeyAlias, shown in the example above), please vote for Take Control of SSH.

Thursday, February 8 2007

Remote Control and Reminders

Frequently, I need to leave myself a reminder or send myself a note on another Mac. I used to do this via email, but now mostly use ssh, with the open and bbedit commands. Since I use ssh private keys for authentication, this is very convenient -- I can issue commands to remote machines without having to enter a password each time.

Since I ssh so often, I've set up several convenience aliases, including ss for "ssh salt" and sc for "ssh cayenne".

For several years I wished for remote clipboard support; I believe Peter Lewis even wrote a tool to implement it, and there are remote control tools such as VNC, but this is simpler and quicker.


Here are some examples of useful remote commands:

Open a BBEdit window on salt, showing the entered serial number:

echo "Serial Number for Some Super Software" | ss bbedit

Open an article in Safari, on cayenne:

sc open http://db.tidbits.com/article/8835

Open a couple pages in Safari on cayenne, including one with punctuation in the URL that would normally trip up the shell:

sc 'open http://blogs.zdnet.com/threatchaos/?p=311 http://daringfireball.net/2007/02/more_crap_from_enderle'

I keep most of my systems on all the time, so it's extremely handy to be able to toss text and URLs around this way.

Serious UNIX nerds can use cat to send a block of text over (Control-D at the end terminates text entry and bundles off the text):

cat - | ss bbedit
This is a test.
This is only a test.
If this was a real emergency, you'd be dead already.
^D

Alternatively, create a text file via a normal ssh session, and open it for later -- this is not as fast, but conceptually closer to normal usage:

pepper@www:~$ ss
Last login: Thu Feb  8 18:13:36 2007 from www.reppep.com
Welcome to Darwin!
pepper@salt:~$ vi Desktop/note.txt
[Type, paste, whatever, here; then save and exit vi]
pepper@salt:~$ bbedit Desktop/note.txt
pepper@salt:~$ logout
Connection to salt.rockefeller.edu closed.
pepper@www:~$ 

Additionally, I do something similar to remind myself to edit or update documents when I get home, which looks something like this:

sc bbedit www/pepper/public_html/index.html

Friday, August 18 2006

ssh sessions hang when adding a higher-priority network connection

I have an MBP which is always connected via AirPort. When I come back to my desk, I typically plug it into an Ethernet cable, on a different (more trusted) subnet.

If I have any ssh sessions open when I bring up the Ethernet connection (which is of course faster and prioritized above AirPort in SP:Network), the ssh sessions hang. If I notice quickly, I can disconnect Ethernet, finish my work (if possible), and disconnect before plugging back into the Ethernet, but this shouldn't happen. The ssh connections are established on the AirPort interface, which is not disturbed, and should remain usable even when an additional connection is introduced.

I have reproduced this behavior on a PBG4 and an MBP, both using the latest

Thursday, May 4 2006

ssh keychain vs. ssh Forwarding

I use Gentoo keychain heavily, to cache ssh private keys. If you use it on a few workstations, and make it a point of clearing keys or stopping keychain, you will find yourself in a situation where you ssh from a machine with keys into a machine without keys. For a while I was picking up the (useless) keychain on the remote machine I sshed into, instead of the loaded keychain on the local machine I sshed from. This .profile refinement doesn't load keychain if it can find an active agent:

  if [[ `which keychain` ]]
  then
    if [[ ! ${SSH_CLIENT} ]]
    then
      keychain -q --noask; . ~/.keychain/`uname -n`-sh
    fi # [[ ! ${SSH_CLIENT} ]]

    alias  kc="keychain --timeout 540 ~/.ssh/id_dsa; . ~/.keychain/`uname -n`-sh"
  fi # [[ `which keychain` ]]

Update 2006/05: I recently switched from Gentoo keychain to SSHKeychain. It optionally integrates with the Apple Keychain (including automatic locking and loading of keys), and configures global environment variables for BBEdit and other co-operating tools.

http://sshkeychain.org/

Saturday, February 4 2006

VNC & ssh through NAT

I periodically need to help my father with his computer. Since he's on DHCP behind an AirPort Extreme (which is getting its own IP via DHCP), this has been tricky. I recently found the solution.

Here's what I sent to Dad:

  • Please go to System Preferences:Sharing:Services.
  • Make sure Apple Remote Desktop is checked.
  • Select Apple Remote Desktop.
  • Check Show status in menu bar.
  • Click on Access Privileges, and make sure VNC viewers may control screen with password: is checked.
  • Type your password into the text box.
  • Hit OK.
  • Repeat this on all your Macs.

Next time I need to access one of your Macs, paste the following into a Terminal window: "ssh -R 6900:127.0.0.1:5900 -R 6922:127.0.0.1:22 www.reppep.com".

After this, I should be able to connect to your computer without any more futzing on your end.

Thanks to RimuHosting for the idea.