Extra Pepperoni

To content | To menu | To search

Thursday, August 14 2008

Suggested iPhone apps

Frank just got an iPhone, so I was listing off suggested apps, and decided to post the list. Almost all of them are free.

  • NetNewsWire/iPhone: RSS reader which synchs with NNW on Mac, FeedDemon on Windows, and Newsgator Online; all are free
  • Instapaper: Multi-computer bookmarking service -- links to http://www.instapaper.com/
  • (paid) Twitteriffic Premium (Free shows ads): http://twitter.com/reppep
  • Stanza: ebook reader
  • Remote: iTunes & AppleTV control
  • (paid) TouchTerm: ssh client
  • (paid) pTerm: ssh client
  • Facebook
  • AIM (just for free SMS)
  • Now Playing
  • Scribble: need a drawing program to play with Julia
  • Shazam: identifies recorded music the iPhone can "hear"
  • Shakespeare: complete works
  • Yelp (Amy likes)
  • Google

Games

  • (paid) Toy Bot
  • Phone Saber
  • Fire Drop
  • Moonlight Mahjong Lite
  • Labyrinth LE
  • Life
  • Tap Tap Revenge
  • Advent (I don't play it, but keep it for the ecstasy it will someday induce in an old Zork fan)

Friday, July 11 2008

iPhone Apps: First Impresssions

I've been waiting for NetNewsWire for iPhone since I first heard of it, and have already registered Twitterrific Premium, which is very slick (although I'm not sure how GPS or photos work). I am somewhat disappointed that NNW/iPhone doesn't proactively download updates; that's one of the nice things about NNW on the Mac -- it's always pretty current, and I never have to wait for an update. On the iPhone, where I may not even be able to get an update on the train, it's problematic. I was hoping NNW/iPhone would proactively sync feeds, so I could use it on the subway while out of coverage, but no joy.

Twice, all apps have failed to launch until I rebooted, and I've had a couple unexpected reboots.

Most apps are very slick, although AIM and iMaze both disappoint. Very much looking forward to using Remote for real, and wondering if I should have gotten an Apple TV for our living room stereo instead of an AirPort Express/n...

There's a trick to replacing the 4 persistent apps in the Dock at the bottom: you cannot drag into the Dock to bump them out of the way; instead you must drag something out of the Dock to make room first, and then you can drag an app into the free space.

It's annoying that deleting an app from the iPhone leaves it on the Mac; moreso that re-synching re-installs the app on the iPhone and forces a full (slow) backup of the iPhone. Adding insult to injury, I cannot control-click an app in iTunes to re-install it, or get rid of the confirmation on every deletion from iTunes.

The AIM client stinks. Not sure if it's push enabled, but it has serious flaws and bugs, both.

Moving apps around Springboard is a bit buggy. As I moved them from one screen to another, Springboard moved a bunch of extra apps to later screens -- many more than were actually necessary to make room. I always have 7 screens of apps, even when they all fit on 6. Under 1.1.4, there were no empty screens -- empty ones were automatically removed; I preferred that behavior.

I expect to get an iPhone 3G Monday -- can't do it this weekend.

Where's the OpenSSH port?!?! I do hope Apple didn't reject a submission...


Update 2008/07/12: The extra screen is correct. When in app rearrangement mode, the iPhone always provides an extra screen so I can move apps there; in normal mode the extra screen goes away.

Friday, February 8 2008

HP c-Class c7000 Chassis & Onboard Administrator Notes

The Onboard Administrators (we got a pair for redundancy) each ship with a unique password. When you connect them, it appears the active OA resets the standby password to match the active. This was a bit confusing, as OA #2 came up active, and the passwords were not as expected; SSL certificates are created and reloaded in terms of "Active" & "Standby", so I initially loaded new certs onto the wrong OAs.

ssh Implementation Flawed

The OAs support ssh access and ssh keys, but apparently only for the single Administrator account. This is documented incorrectly -- the docs say the last word on the key line is the username the key is for, but actually they're all linked to Administrator. HP Support doesn't know much about it. It's bad when security features don't work as documented -- in this case, it would be easy to follow instructions and upload a key for an unprivileged Operator or User account, unintentionally granting full Administrator access -- we had this for a while, until I figured out what was really going on.

The web interface doesn't allow copy & paste of keys -- they must be downloaded by the OA from a web server. Afterwards, though, the public keys (which had to be accessible on through a web server, remember) are not visible to other authorized users of the OAs -- only Administrator can see or modify keys. Feh.

Additionally, the web interface shows line breaks as '^', so the keys look corrupt. Despite this they work, and display correctly in the command-line interface.

OA doesn't automatically configure its accounts onto blade iLO. Instead, it creates an account for OA itself on each blade's iLO. This is a bit odd, as it means authorized users cannot connect directly to iLO -- instead they must connect through an OA, and have the OA login, before using iLO. We will presumably use the Compaq iLO configuration language to deploy our accounts to iLO, but this shouldn't be necessary.

Good News

On the bright side, the chassis is easier to mount than our (smaller) IBM BladeCenter chassis; it's also better labeled. The Onboard Administrator interface is better laid out, although it doesn't work in Safari (seems fine in Firefox/Mac). The command line is a bit less bizarre than IBM's.

HP makes it easy to dump the configuration to a text file, tweak it, and load it into another chassis, although we haven't tested yet; they call this "Configuration Scripts".

Thursday, January 24 2008

Keychain Sync without .Mac

After getting burned too many times, I dropped my .Mac subscription. I never trusted my Apple keychains to iDisk anyway, but this means I have different subsets of passwords on different machines, and no good way to keep them in sync. I thought of a solution for manual sync last week: One keychain per Mac. Say I have 3 systems: work, home, and other. Each system has 3 Apple keychains: work.keychain, home.keychain, and other.keychain, with each host using its own as the default. Then I can rsync work.keychain to home.keychain & other.keychain, etc. This is awkward with rsync because it's inherently unidirectional, but keychains are small so it's quite feasible to script.

In Tiger, I know the keychain is actually stored in memory once it's unlocked, so it's good to lock (unload) all keychains with "security lock-keychain -a" before updating the files -- this goes in the same script. I also set mine to lock after 2 hours of inactivity, or (on those systems where I run SSHKeychain) when sleeping or activating the (locking) screen saver.

Saturday, December 1 2007

Yay! Leopard fixed kickstart

ARD includes a very handy script called kickstart (/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart), to configure the Remote Desktop agent, which is also what Leopard's Screen Sharing uses. This is important because Murphy says that you will always first need to connect to a recently installed machine and only then discover the ARD agent is off. With the kickstart agent, you can configure user access to Remote Desktop through an ssh connection, and turn the agent on.

Unfortunately, it never worked for me. I have tried to use kickstart on at least 4 separate occasions (always on Tiger systems), and it never did what I wanted. Tonight, I used it on a 10.5.1 system, and in about 5 minutes I had access (manually tunneled through ssh, no less). It would have been faster if the kickstart command was simple (it's somewhat involved), or if I wasn't determined to configure access controls before turning on ARD. It's easy to configure ARD access via System Preferences:Sharing, but bad practice to enable services without access control configured.

Hoo-rah!

To learn about kickstart, use sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -help. If WordPress won't let you read that whole line, try copying it into another program. Apple's Apple Remote Desktop Administrator’s Guide includes some helpful examples.

We also use an UID 0 account, which doesn't appear in System Preferences:Sharing, so I tend to create the account, set the UID, remember ARD, and curse as I discover I can no longer enable ARD access to that account without restoring the UID -- quite a nuisance. Since local accounts are now stored in .plist files, adding our UID 0 account and giving it ARD access should both be much easier now.

Saturday, November 3 2007

Screen Sharing replaces Apple Remote Desktop

Update 2009/01/15 If you connect to a particular machine frequently, you could put a clickable icon into the Dock.

  1. Put these two lines into a plain text file (I'll call it myserver.command). The filename must end with .command to be launchable from the Finder.
  2. Make sure it has UNIX line breaks.
  3. Make it executable (chmod +x myserver.command).
  4. If you use it a lot, drag it into the Dock for quick access.
#!/bin/sh
(sleep 4; open vnc://127.0.0.1:5901) & ssh -C -4 -L 5901:127.0.0.1:5900 myserver

That will ssh to myserver, pop back a tunnel for VNC, and point Screen Sharing to the tunnel. After you close the Screen Sharing connection and log out of the ssh session, the tunnel will be closed automatically.


Update 2008/2/3: Adam, thanks for the suggestion -- I'd forgotten about the vnc:// scheme. But who's Geoff?? I prefer aliases to functions because they're simpler, and like to leave an ssh shell open, both for my own use and as a reminder to close the tunnel when done. Here's a simpler alias -- note that you must still supply the hostname on the command line after the alias, e.g., "stss salt".

alias stss="(sleep 4; open vnc://127.0.0.1:5901) & \
ssh -C -4 -L 5901:127.0.0.1:5900"

Update 2007/12/14: I added a pbcopy command to put '127.0.0.1:5901' on the Clipboard (pasteboard), so now I can just Paste and then delete (pbcopy appends an undesired Return to the Clipboard), which makes the whole thing easier. New alias (note that this is really properly one line, but it doesn't wrap properly without help):

alias stss='echo 127.0.0.1:5901 | pbcopy; open \
/System/Library/CoreServices/Screen\ Sharing.app; \
ssh -C -4 -L 5901:127.0.0.1:5900'

I have a couple licenses for Apple Remote Desktop at work, for managing our 8+1 Mac cluster ("the orchard") and for managing other Mac servers on campus. I find ARD very useful because although Remote Desktop uses VNC as the underlying protocol, Apple's compatibility has been poor, so I had lots of trouble connecting from Chicken of the VNC and other clients. While I like ARD (particularly the automatic ssh tunneling in v3), I only use the remote control feature, never its other management capabilities.

With Mac OS X 10.5 Leopard, Apple has bundled /System/Library/CoreServices/Screen Sharing.app, which provides the VNC capabilities I use from ARD and skips the other features I don't care about. It's my favorite Leopard feature, accessible from the Finder Sidebar, iChat, Server Admin, and through Back to My Mac (which seems to have some problems with security).

The only thing I don't like about Screen Sharing is that Apple apparently built encryption into the VNC protocol in an incompatible way. Apple's encryption is of course incompatible with all the other clients & servers, since it's Apple proprietary (just like their proprietary compression encodings). It's confusing because the Preferences options look identical to the ones in ARD3, which actually uses an ssh tunnel to provide encryption. It's a firewall problem because there are lots of places we a) allow ssh, b) block unencrypted VNC, and c) would allow encrypted VNC. ARD3's ssh tunneling is usable here but Screen Sharing's port 5900 connection is blocked. Fortunately the workaround is simple -- build the ssh tunnel manually, as is normal for non-ARD3 VNC users. I have this alias:

alias stss='open /System/Library/CoreServices/Screen\ Sharing.app/; ssh -C -4 -L 5901:127.0.0.1:5900'

I use it with a hostname, as in: stss www

That makes an ssh connection to the specified host (www in this case), sets up a tunnel from 5901 on my admin workstation to 5900 on the server (since the admin workstation is likely to be running the Remote Management/Screen Sharing agent on 5900 already), and gives me a shell on www. As a convenience, it also launches Screen Sharing for me. In the Screen Sharing Connect window, I type 127.0.0.1:5901, and connect to the local end of the tunnel on port 5901; it goes through ssh and I get secure remote control via the ssh port (so it works across any firewalls that allow ssh). It's actually doubly encrypted if I'm going across the Internet, since I always leave Screen Sharing's encryption on too -- if I forget to start the tunnel or connect to a machine that's not firewalled on port 5900, I want to be sure I'm not transmitting passwords in plaintext.

Friday, November 2 2007

Mac OS X: Authentication Timers

I have been reading about Mac OS X 10.5 Leopard Server and non-Server lately, and I was surprised to realize how many different authentication & authorization systems are running each with its own timer.

  1. Access to the "console" (keyboard/video/mouse): Ends when you log out or the locking screen saver kicks in.
  2. Authentication for administrative actions in Carbon/Cocoa programs (such as modifying system directories in the Finder): 5 minutes (I believe).
  3. Apple Keychain: Doesn't lock automatically unless you configure it in Keychain Access.
  4. ssh-agent (now linked to the Apple keychain): Clears when you reboot or when the Apple keychain locks.
  5. Kerberos V (both client-server and client-client): Apparently TGTs expire after 10 hours by default.
  6. sudo: 5 minutes by default.

Tuesday, August 21 2007

SSHKeychain Is Unsafe

Eric Warnke has discovered that, if asked nicely, SSHKeychain will print out the passphrase used to encrypt a loaded private key. This is bad, as the whole point of an ssh agent/keychain is to provide secure access to encrypted keys, meaning you cannot get the passphrases or plaintext keys out.

http://www.sshkeychain.org/pipermail/users/2007-August/000098.html

Crud on a cracker, Batman!

Verified in the latest (v0.8.1) -- hopefully there will be a patch soon, but this just shouldn't be possible.

http://www.sshkeychain.org/

Friday, August 3 2007

Laptops in Hostile Environments

The Register has an interesting article on how to "safely" take your computer to Defcon, with the very wise suggestion that you're safer if your laptop does not go to Defcon. Cellular phones without 802.11 are probably okay for this year at least. They refer back to a much more hard-core SANS post on the same topic.

The exercise is more involved for the fully paranoid, or generally when preparing to enter a truly hostile network. I assume that someone at Black Hat/Defcon has an unannounced exploit that I'd be vulnerable to. This implies you shouldn't have any sensitive data or access to sensitive machines. Since you wouldn't need a laptop without data or access, you probably need to mitigate the consequences of getting hacked.

  1. Make up a couple disposable passwords just to use at the conference, one for this machine and one for outside accounts. Destroy them later.
  2. Bring an empty USB thumb drive.
  3. Create a new email account, so you can send yourself notes/presentations for later.
  4. Forward your important email accounts to the new one (keep copies on the normal accounts), so you don't have to check them.
  5. Note that if you have a hosting plan like DreamHost's, you can create brand-new ssh and email accounts free. I believe DH offers SSL webmail, if you can ignore the certificate warnings.
  6. Get a cheap monthly VPN account, as suggested by Glenn Fleishmann; this is much simpler than establishing a trusted Squid proxy on a non-sensitive box, as originally suggested by SANS; note that you are then trusting the VPN provider.
  7. If you have any untrusted protocols, try to access them from your temporary shell account via ssh, or though an ssh tunnel.
  8. Back up your data (image the drive -- you will need it later, and a full image is fastest to restore).
  9. Wipe your laptop.
  10. Install your OS, creating a new account with your new local password.
  11. If you have a built-in webcam with an independently software-controlled active light, tape over it. If you feel comfortable opening up your laptop, disconnect its internal microphone.
  12. Create a new ssh keypair. If you know the netblock, only allow access from Defcon machines and your own personal host(s); I have some info on doing this in authorized_keys in Take Control of SSH, Draft Excerpt: Public Key Authentication. Make sure your laptop trusts your other (home) machines.
  13. Only as needed, trust this new key on your other systems.
  14. In your local firewall, block outbound access except to the ports you intend to use; this is easy in Linux, but a bit more complicated on Macs, where you need to write your own startup script (or .command script in Login Items). This is obviously overridable, but an effective way to make sure you don't accidentally connect without encryption, either from habit or because a website redirects you to unencrypted HTTP to save encryption cycles (common). For services where you know what host you'll be connecting to, embed that. Here's a sample of what you might add to add Apple's ipfw. Note that it's easy to shoot your own foot off with outbound firewall restrictions.

    ipfw add 01010 allow tcp from any to 4.2.2.1 dst-port 53 out
    ipfw add 01020 allow udp from any to 4.2.2.1 dst-port 53 out
    ipfw add 01030 allow tcp from any to 4.2.2.2 dst-port 53 out
    ipfw add 01040 allow udp from any to 4.2.2.2 dst-port 53 out
    ipfw add 01110 allow tcp from any to any dst-port 22 out
    ipfw add 01120 allow tcp from any to any dst-port 443 out
    ipfw add 01030 allow tcp from any to smtp.dreamhost.com dst-port 587 out
    ipfw add 01040 allow tcp from any to mail.dreamhost.com dst-port 993 out
    ipfw add 01900 deny tcp from any to any out
    
  15. Restore only required information to your laptop.


Enjoy the conference. Hi, Rich!


When you're back home, connect from your home machines to the untrusted laptop, rather than the other way around, retrieve any data on it, and then boot from CD/DVD/PXE and reinstall, or restore from your image if you can do that without using the untrusted OS on the laptop's hard drive.

Monday, July 16 2007

My iPhone Dilemma

I want an iPhone, of course.

My problem (aside from cash outlay to purchase -- the monthly compares well to my current Verizon unlimited data plan), is that it's not quite what I want.

I have a 60gb iPod photo, which has been full almost since I got it. I stopped using lala a while ago because the iPod can't hold any more music, and I really want it all with me. Ironically, I can't listen to my iPod most of the time, since it's usually plugged into a Mac with my Eudora Folder mounted. On the other hand, using rsync, I keep all my music on all my Macs, so the iPod being unavailable is not a big problem.

I use my Treo 650 very heavily, for the following (roughly prioritized):

  1. Phone: less than 30min in a typical day, but critical.
  2. SMS Pager: light use, but critical.
  3. Plucker: anywhere from 10 to 180 minutes per day.
  4. Address Book & Calendar, many times a day, but iSync keeps corrupting my data.
  5. TCPMP to watch TV: anywhere up to 120 commuting minutes per weekday; unfortunately TCPMP can't handle H.264 video quality, and capacity is limited to an SD card. I think my 2gb SD card is the biggest it can handle, but cannot confirm; I'm not going to carry a bunch of SD cards around to watch more TV.
  6. Web Confidential: Apple's 4-digit PIN is completely inadequate to replace this.
  7. TomTom Navigator: indispensable on trips, and Google Maps doesn't support Bluetooth GPS.
  8. Vindigo: Replaceable by Google & Google Maps, and much less important now that I'm a father. ;)
  9. Documents to Go: iPhone can handle this directly.
  10. Web browsing: I rarely do this over Verizon's network.
  11. Still & video camera: The Treo's stinks, and I carry my Canon SD800IS during weekends.
  12. Salling Clicker: recently replaced by an Apple Remote.
  13. TuSSH/pssh: I'd really really really like an iPhone ssh client with secure private key storage, but the Treo applications are not mature either. Instead I now carry a laptop to all meetings, which is a significant productivity booster, but much bigger and more expensive than an iPhone.
  14. Games: I rarely have time for them, and the iPhone will certainly get some, as it's a much better platform than the iPod.

In contrast, I use my iPod anywhere from 0-120 minutes per weekday for listening to music; the rest of the time it's generally mounted as a hard drive. If and when Thunderbird + Penelope is suitable, I'll be delighted to switch to independent IMAP clients on all my Macs.

So what I want is an iPhone with a 100gb drive, ssh client with encrypted private key support, encrypted data storage, the ability to read Safari web archives (Safari on Mac & Windows would need the complimentary ability to create such archives, like Plucker), and GPS support in Google Maps.

Without a hard drive, it's not a suitable replacement for my iPod, and I will be quite surprised if Apple doesn't offer a 100gb large-screen iPod sans phone by the end of this year.

Without offline browsing, encrypted data storage, and GPS, it's not a good replacement for my Treo 650.

I hope the next-generation iPhone includes 16gb, 3G celluar data, and GPS. If I got a current model iPhone, I'd probably keep 2-3gb of video and 3-4gb of music on it, and deal with the frustration of not having the rest of the music. I really do not want to switch to watching TV instead of reading (computer) news on the train, but I could console myself with surfing as I walk to and from the train. I'd probably bring my Treo on trips for GPS. I'm not sure what I'd do about Web Confidential. Most of the time I could get the same data from a Mac at home or work, but not always. I could put a subset of the data on an SSL and password protected web page for reference, but that's risky.

I currently carry noise-blocking stereo earphones, a phone headset, and a 2.5mm-3.5mm adapter; I switch back and forth a couple times a day. A single high-quality headset with mic and button sounds great, although I'd prefer an in-ear model with more noise blocking than Apple's In-Ear Headphones, which I found inferior to both Shure and UltimateEars 'phones.


Update: I have an iPhone for a couple days on eval, and I've confirmed that although it will accept my WPA Enterprise password, it can't actually join the network. This is a problem as we only allow access to many management interfaces from trusted networks, and the wireless one requires WPA Enterprise.

Wednesday, July 11 2007

.Mac synchronization, the free way (rsync)

Hooray! I now have a functional script for copying my iCal, Address Book, and Safari bookmarks. I suspect it's copying more than necessary, so I may refine it, but it's doing what I need now. I am amazed at how many places Apple sticks calendar data!

A big thank you to the rsync and OpenSSH developers. Once again, the script is largely comments, but that's good -- it is fundamentally simple.

Here is version 1.0 -- please check http://www.extrapepperoni.com/category/computers/synchronization/ and http://www.reppep.com/~pepper/code/ to see if there's a newer version.

I wrapped the lines here to make it more readable.


#!/bin/sh
# rsync-dotmac.sh
# v1.0, 2007/07/11 by Chris Pepper
# Check for a newer version at:
# <http://www.extrapepperoni.com/category/computers/synchronization/>
# <http://www.reppep.com/~pepper/code/>

for h in mac1 mac2
 do
  ssh $h rm -Rf ~/Library/Caches/com.apple.iCal ~/Library/Caches/iCal \
   ~/Library/Caches/Metadata/iCal \
   ~/Library/Preferences/com.apple.iCal.alarmsCache.plist \
   ~/Library/Preferences/com.apple.iCal.AlarmScheduler.plist \
   ~/Library/Preferences/com.apple.iCal.plist \
   ~/Library/Preferences/com.apple.iCal.sources.plist \
   ~/Library/Preferences/com.apple.iCalSync.AlarmScheduler.plist
  rsync -va --delete ~/Library/Application\ Support/AddressBook/ \
   $h:"Library/Application\ Support/AddressBook/"
  rsync -va --delete ~/Library/Application\ Support/iCal/ \
   $h:"Library/Application\ Support/iCal/"
  rsync -va --delete ~/Library/Calendars/ $h:Library/Calendars/
  rsync -va --delete ~/Library/Safari/Bookmarks.plist \
   $h:Library/Safari/Bookmarks.plist
 done

# To use, enter your read-only hostnames on the 'for' line above.
# They must, of course, be accessible from the system you run this script on.
# The script pushes data (Address Book, iCal, and Safari bookmarks) from
# a master to one or more read-only systems. It is intended as a free
# replacement for a part of Apple's .Mac service that caused me some
# frustration.
# I wish Apple provided a way to tell those applications not to allow
# (futile) changes on read-only systems, although with .Mac services
# that should not be necessary.
# I don't sync ~/Library/Keychains/, because my keychain items vary
# between systems, so unidirectional sync would be suboptimal.
# You may get warnings when going between OS versions; I have had no
# trouble just ignoring them (so far).
# If you get double birthdays, try disabling and re-enabling the
# Birthdays calendar in iCal's Preferences.

Wednesday, June 6 2007

DreamHost Screwed the Pooch

This sucks, Beavis!

I had to change the school's panel password (which will only be changed again in 24 days when I hand over the reins to my successor, who will change it again), plus my personal panel password, 3 shell passwords, 12 list passwords, this blog's password, and possibly the Analog passswords, and what will I forget?

Crud. Crud. Crud.

DreamHost's next failure is not telling us what the spam links look like. I can't read the source of every page on the site, but if they would tell us what to look for, I could grep for the suspect sites quite easily. I actually found them elsewhere (see the Andy Hagan link below) -- no thanks to DH, whose screw-up I'm now attempting to fix.

Adding injury to injury, I just got the cheery/goofy montly DreamHost email, with no mention of the hack, even though they must have been dealing with the break-in when the message was sent.

To make things worse, the status page where they promise to post updates on this incident doesn't even mention it! This is 2 1/2 hours after they sent me email, and they still haven't come clean in public. On the other hand, oscandy.com had a posting about it a couple days earlier http://www.oscandy.com/hacking/454-dreamhost-hosting-platform-hacked.

From a couple other blogs, this may have shown up a week ago:


To: "Chris Pepper"
From: DreamHost Security Team
Subject: [reppep 11988754] URGENT: FTP Account Security Concerns...
Date: Tue,  5 Jun 2007 18:52:42 -0700 (PDT)

Hello -

This email is regarding a potential security concern related to your  
'reppep' FTP account.

We have detected what appears to be the exploit of a number of  
accounts belonging to DreamHost customers, and it appears that your  
account was one of those affected.

We're still working to determine how this occurred, but it appears  
that a 3rd party found a way to obtain the password information  
associated with approximately 3,500 separate FTP accounts and has  
used that information to append data to the index files of customer  
sites using automated scripts (primarily for search engine  
optimization purposes).

Our records indicate that only roughly 20% of the accounts accessed -  
less than 0.15% of the total accounts that we host - actually had  
any changes made to them. Most accounts were untouched.

We ask that you do the following as soon as possible:

1. Immediately change your FTP password, as well as that of any other  
accounts that may share the same password. We recommend the use of  
passwords containing 8 or more random letters and numbers. You may  
change your FTP password from the web panel ("Users" section, "Manage  
Users" sub-section).

2. Review your hosted accounts/sites and ensure that nothing has been  
uploaded or changed that you did not do yourself. Many of the  
unauthorized logins did not result in changes at all (the intruder  
logged in, obtained a directory listing and quickly logged back out)  
but to be sure you should carefully review the full contents of your  
account.

Again, only about 20% of the exploited accounts showed any  
modifications, and of those the only known changes have been to site  
index documents (ie. 'index.php', 'index.html', etc - though we  
recommend looking for other changes as well).

It appears that the same intruder also attempted to gain direct  
access to our internal customer information database, but this was  
thwarted by protections we have in place to prevent such access.  
Similarly, we have seen no indication that the intruder accessed  
other customer account services such as email or MySQL databases.

In the last 24 hours we have made numerous significant behind-the- 
scenes changes to improve internal security, including the discovery  
and patching to prevent a handful of possible exploits.

We will, of course, continue to investigate the source of this  
particular security breach and keep customers apprised of what we  
find. Once we learn more, we will be sure to post updates as they  
become available to our status weblog:

      http://www.dreamhoststatus.com/

Thank you for your patience. If you have any questions or concerns,  
please let us know.

- DreamHost Security Team

Sunday, June 3 2007

Hardcore: Installing Mac OS X via Command Line

My old desktop (PMG5 2x2GHz) is now at home, and I am installing Mac OS X Server 10.5 "Leopard" on it. I've done it at least half a dozen times already, sometimes finding bugs and often finding that I want to redo it for one reason or another, typically having to do with rearranging our home network. This is all since the latest beta was seeded, so I haven't had to reinstall to upgrade the beta, although that will happen as well.

When you boot the Mac OS X Server DVD, in addition to showing the normal graphical installer, it additionally starts sshd and Apple's ARD (VNC) server. You can log in via ssh as root with the machine serial number (first 8 characters, although the documentation incorrectly says 'digits'). I haven't tested non-Server, but for heavy-duty workstation deployments you'd probably want to be cloning images anyway, and Apple provides different tools for that. I did a bunch of seaerching for details on the procedure, but the only thing I found is Apple's CLI admin guide.

http://images.apple.com/server/pdfs/Command_Line_v10.4_2nd_Ed.pdf

Unfortunately, Apple does not make their Server Admin Tools compatible with the previous version of the OS, so in a single-test-system scenario, we're stuck with running these nice client-server admin clients on the server. I mostly do it via ARD rather than sit in the "server room" in our basement after the initial install. Better if I can pop the DVD in, reboot, and do everything remotely. Since I'm using a third-party hacked firmware on my Linksys WRT54G which provides static DHCP, I get to skip the step of finding which IP the PMG5 acquired (Apple provides a tool for this, or an nmap ping scan would work if everything else was static) -- DHCP automatically assigns the correct IP, which is also listed in /etc/hosts. Now I am starting to work through the CLI installation process, which resembles the documented Tiger Server CLI installation process.

Unfortunately, my notes on the process have to remain private, since Leopard is under NDA.

Sunday, April 15 2007

Major Mac Movements

I was discussing this week's plans for computer rearrangements with Amy, and was amazed when I listed them all. Most of these will happen this week.

  1. I'm getting a Mac Pro at work; it has shipped and should be in soon.
  2. I got the (very nice) 23" Apple display already; I'll connect it as soon as I get the Mac Pro.
  3. I'll bring my current PMG5 home; it will become a Leopard testbed, and after Leopard's release will become www.reppep.com.
  4. The PMG5 has a 17" Apple LCD with ADC connector, which will come home with it (I don't have anything else at work with an ADC connector, and nobody wants a 4-year-old 17" display).
  5. I bought a 20" LCD at work a few months ago, but the ATI video card Apple shipped with the PMG5 can't drive it and a 17" display properly. I got a replacement card, thinking it was defective, but the (expensive) replacement part had the exact same problem, so it's a design flaw with that model. I brought the 20" display home and brought my own personal 19" display to work. I will bring the 20" back to work.
  6. I will bring the 19" LCD back home (I'll miss the pixels for iPhoto, but otherwise it's fine).
  7. Amy now suddenly needs a new computer (preferably one which can run Windows), and our finances have just gotten tight again, so I have deferred my purchase, and instead bought her a MacBook, which should arrive this week.
  8. At work, I have an original MacBook Pro which I use for a) ssh, b) Safari, c) Leopard testing, & d) Parallels/VMware hosting & testing. I'll bring my own PBG4 in for ssh & Safari, do Parallels/VMware on the new Mac Pro, and move Leopard testing to the PMG5 at home.
  9. I'll bring the MBP home, where it will be much faster than the PBG4.
  10. I'm replacing our 100mbit switch (the old gigabit switch died a year ago) with a new 8-port gigabit switch -- for $35 ($50 before rebate!!!).
  11. I'm running Ethernet to the loft where my desk and printer are; this will free up an AirPort Base Station currently connecting the printer to our home LAN via WDS.
  12. I will replace our upstairs DVR with our hacked Series 1 TiVo, so I can once again extract video to watch on the subway with TCPMP; I will use the newly-freed-up ABS to connect the TiVo's Ethernet.
  13. My 60gb iPod photo should be back soon, so I'll be moving my Eudora Folder (email) back off the old 10gb (which was mine, then Amy's, then attached to the stereo to share).

Then later this month, we'll pack up our offices and move everything to the Super-Tent. I'll be moving the Mac Pro and PBG4 w/ 2 displays, and getting rid of my Sun Blade 100, Dell Windows PC, & Microway Linux PC -- replacing them with VMs on the Mac Pro.

Wednesday, April 11 2007

Backups and Moving into Subversion

I do not have a comprehensive backup strategy. I keep reading well-reasoned arguments for at least 3 complete (bootable) backups, scratching my head, and saying, "Yeah, if I had a couple thousand unused, I could do that." Yes, I know drives are cheap, but I have 3 important systems to back up, and 4 more where perhaps a dozen files matter combined. Between the important systems, I'd need about 600gb usable, and Joe thinks you need 3 bootable backups, and I'm a strong believer in keeping the backups permanently away from the originals (not on a rotating cycle, but never-the-twain-shall-meet time -- what else is broadband for?).

http://www.takecontrolbooks.com/backup-macosx.html

Unfortunately, whenever I kick off an rsync job to update my remote backups (one set, not 3), Amy complains every day because Internet access is so slow. I think I have throttled it back enough to be unintrusive, but I just don't remember, and (especially at an unobtrusive fraction of our 384kbps DSL uplink), it just takes so dang long.

http://rsync.samba.org/

But I'm not too worried, because 95%+ of my important data is in Eudora on my iPod (yes, I know iPods die -- I've killed more than my fair share, along with Zip disks, Orb disks, and even a 2.5" FireWire drive), and I use Synchronize Plus X to back up my iPod to my primary home and work machines automatically, every time I plug it in. This is a great feature of Synchronize, which mitigates my annoyance with their licensing.

Every morning when I get to work, I dock the iPod; Synchronize sees it, launches Eudora, backs up changed files to my work desktop's hard disk, and launches an AppleScript to bring up a couple applications. When I'm done, I type eject in a Terminal window, my iPod unmounts, and login.keychain is locked (flushing ssh keys in the process). On my home PowerBook, the process is the same.

So I know my backup system works well, because I test restores whenever there's a problem with the iPod (perhaps twice a year). But I still feel nervous erasing it before sending it away. I actually did this today -- a couple months after its 2-year anniversary, my 60gb iPod photo's battery won't last an hour. Apple told me they'd fix it under extended AppleCare, but changed their minds and cancelled the repair -- without telling me. I paid the $70 and sent it in today; I'm running off our old 10gb iPod in the interim.

For the next step, I'm moving my website into Subversion. I already keep Julia's school site & Bjorn's site & documentation in Subversion, and last week I got sick of making manual backups of a piece I'm writing for TidBITS, so I reorganized and checked in my writing (not that there's much of it). Tonight, I threw out 4gb of my 7gb ~/public_html in preparation for checking in (most of) the rest of it.

Backing up a Subversion repository is easy, and will bring me to about 99% of important data backed up (not counting MP3s -- are already replicated to 2 Macs & an iPod -- or iPhoto libraries, which require different handling, but should back up quite well via rsync; but I have to get Julia's few [important] pages into Subversion too).

Tuesday, April 3 2007

Today's Accomplishment: SSO ssh Upgrades

Today I upgraded our 8 core authentication ("Single Sign On") servers from commercial ssh to OpenSSH (it was a 2-year battle to standardize on OpenSSH, but in the end the right product won).

http://www.openssh.org/

The tricky things are: a) These machines are critical enough that they're not directly accessible from campus, so everything must be done through an intermediary machine. This complicates everything. b) Upgrading ssh is problematic because it's the remote control tool, so working on sshd implicitly interferes with your own control. Fortunately we have good terminal servers, which make this much less problematical; when sshd is down, you can get in the back door to bring it up.

One of the neat things about UNIX is that you can delete a file, but if it's open the file is not actually deleted until that filehandle is no longer in use (when the last filehandle is closed, the disk space is reclaimed), so for non-terminal-server systems, I've actually done the whole upgrade through an sshd binary which is deleted at the beginning of the upgrade, replaced during the upgrade, and still in use until the very end.

I (we) have been doing this long enough, and refined the procedure sufficiently, that the whole upgrade took under 90 minutes total for 8 machines, although there was a lot of prep and follow-up, cleaning up accounts, /etc/sudoers, installing public keys, etc.

Overwriting /etc/passwd, /etc/shadow, and /etc/groups with copies from the intermediate machine was particularly stressful; I was highly relieved that nothing broke.

Saturday, March 24 2007

Take Control of SSH, Draft Excerpt: Public Key Authentication

I've started writing something I hope will be an ebook called Take Control of SSH. It's an amazingly long process, so here's an excerpt on private and public key authentication, the most important part of SSH. Note that this was not written as a stand-alone piece, so please excuse the rough edges.

If you'd like to read the rest, please vote for Take Control of SSH.

Part II: Public Key Authentication (Practice)

Private keys are much more secure than passwords (which must be short enough for people to remember and easy to type), but fundamentally they're still large numbers which function as password equivalents. This means that anyone who has your private key can use it to authenticate as you to an SSH server. Files containing private keys (such as ~/.ssh/id_rsa) are therefore as security sensitive as passwords.

On the server side, public keys serve identify users to the SSH server. If someone can manipulate your ~/.ssh/authorized_keys file, they can insert their own key and impersonate you. To protect against this, sshd ignores authorized_keys files with insecure permissions. Unfortunately, sshd doesn't complain about this in a visible way, so mysterious failures of public key authentication are often traced back to inadequate permissions. The rule is that the ~./ssh/authorized_keys file and all the directories that contain it, up to the root of the filesystem, must not be writable by any account except the owner. File system permissions cannot stop a system administrator from accessing the key files (both private and public, although the private keys should be encrypted), of course. Similarly, anyone with access to the machine or unencrypted backups could copy the key files.

Create a Public/Private Keypair

To get started with public key authentication, first generate a new rsa key in Terminal with "ssh-keygen -t rsa". Let ssh-keygen create ~/.ssh/id_rsa, and provide a good password you will remember, because keys cannot be recovered if you forget your password. ssh-keygen will create ~/.ssh if it doesn't already exist, as well as ~/.ssh/id_rsa (your new private key, encrypted with the provided passphrase) and ~/.ssh/id_rsa.pub (your new public key).

pepperbook:~ sample$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/sample/.ssh/id_rsa): 
Created directory '/Users/sample/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/sample/.ssh/id_rsa.
Your public key has been saved in /Users/sample/.ssh/id_rsa.pub.
The key fingerprint is:
ba:d1:0c:43:93:a8:d5:5a:b8:a2:4b:29:ae:34:ee:3f sample@pepperbook.reppep.com
pepperbook:~ sample$ ls -ld .ssh
drwx------   4 sample  sample  136 Apr  6 00:22 .ssh
pepperbook:~ sample$ ls -l .ssh
total 16
-rw-------   1 sample  sample  736 Apr  6 00:22 id_rsa
-rw-r--r--   1 sample  sample  618 Apr  6 00:22 id_rsa.pub
pepperbook:~ sample$ 

To use the new public key for logging into this account on this machine, copy id_rsa.pub to authorized_keys: "cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys". Test it with "ssh 127.0.0.1". First, you will be asked to accept the public key for 127.0.0.1 (recall that keys are used to identify servers, as well as users); it's then automatically copied into ~/.ssh/known_hosts. Second, you will be asked for the passphrase for your private key, instead of the account password for the remote system (note that the passphrase might be the same same as the account password). Finally, you will get a UNIX shell dmg pt on the "remote" system once the ssh session is established. Log out to terminate the ssh session.

Warning: Some Mac OS X versions suffer confusion between the current IPv4 Internet Protocol and the newer IPv6 protocol; IPv6 enables more computers to connect to the Internet but is not yet in wide use. To avoid these issues, use 127.0.0.1 in commands instead of localhost. They should be equivalent, but are not (at least in early versions of Mac OS X 10.4 "Tiger", and I believe in "Panther" as well). Fortunately, Apple fixed this by 10.4.8, but it caused a lot of confusion before that, and still does for people using older versions.

pepperbook:~ sample$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
pepperbook:~ sample$ ssh 127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is 8a:d3:22:82:f5:2b:88:f0:20:1b:72:bd:aa:30:e0:e2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.
Enter passphrase for key '/Users/sample/.ssh/id_rsa': 
Welcome to Darwin!
pepperbook:~ sample$ ls -l .ssh
total 32
-rw-r--r--   1 sample  sample  618 Apr  6 00:23 authorized_keys
-rw-------   1 sample  sample  736 Apr  6 00:22 id_rsa
-rw-r--r--   1 sample  sample  618 Apr  6 00:22 id_rsa.pub
-rw-r--r--   1 sample  sample  219 Apr  6 00:23 known_hosts
pepperbook:~ sample$ exit
logout
Connection to 127.0.0.1 closed.
pepperbook:~ sample$ 

Now that you've confirmed your public and private keys work, it's time to copy authorized_keys to some other SSH servers. First, make sure there's a suitable ~/.ssh directory, next send the file, and then login using the key to confirm it works.

pepperbook:~ sample$ ssh www "mkdir -p .ssh ; chmod 700 .ssh"
The authenticity of host 'www (66.92.104.200)' can't be established.
RSA key fingerprint is 53:66:e9:b5:92:e1:5f:d9:71:fa:87:7b:35:99:f2:d3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www,66.92.104.200' (RSA) to the list of known hosts.
Password:
pepperbook:~ sample$ scp ~/.ssh/authorized_keys www:.ssh/
Password:
authorized_keys                               100%  618     0.6KB/s   00:00    
pepperbook:~ sample$ ssh www
Enter passphrase for key '/Users/sample/.ssh/id_rsa': 
Welcome to Darwin!
www:~ sample$ exit
logout
Connection to www closed.
pepperbook:~ sample$ 

Once that's tested, you're ready to copy authorized_keys to more computers. You can also put the id_rsa file on other machines, but be careful with this file! Don't put it on any machines you don't completely trust.

Remember that you can use multiple private and public keys. The id_rsa and id_rsa.pub filenames are just defaults -- keys don't need to have any special names, although agents tend to assume that the public key filename is the private key filename plus ".pub", and may get confused otherwise.

SSH Authentication Agents

So far you've replaced password authentication with public key authentication. This is valuable, because it means someone who controls an sshd server which you log into will not get a chance to steal your password. The way SSH handles plain password authentication (against the built-in UNIX or Windows account databases) is by taking the password typed in and sending it through the ssh connection; at the other end the username and password are decrypted and checked against the remote account's password on file. Both the local ssh program you run and the remote sshd program work with the unencrypted password. There have been major incidents in the past where a popular site (such as SourceForge.net) was broken into, and the crackers recorded all the usernames and passwords used to ssh into SourceForge until people noticed something was wrong. The crackers then used those username/password pairs to log into other sites. It was a colossal mess. Public keys avoid this risk, because the remote system only gets the public key. It doesn't need the private key, and cannot determine it from the public key.

Warning: It is possible to use unencrypted private keys, but this is generally a very bad idea, because anyone who gains access to the private key file (trivial, with physical access to the computer) then controls that identity, and can impersonate or eavesdrop on the owner. About the only time unencrypted private keys are appropriate is for automated tasks, run when nobody is available, if using an agent is not feasible. Such keys should be restricted as much as possible in authorized_keys, and the accounts they access should be restricted as much as possible as well.

Key agents make use of SSH much more efficient, at the cost of a small reduction in security. Agents read encrypted key files; the agent keeps the unencrypted key in memory (never in a file on disk), making it immediately accessible without further decryption or password entry. This makes use SSH dramatically more efficient. Without an agent, connecting to 5 systems 3 times each would require typing one's password correctly 15 times; with a loaded agent, all the connections are automatic. As an example, consider checking a file on several machines. An SSH agent automatically provides the private key for each connection, making large-scale operations quick and convenient.

The risk in using an agent is that anyone who can control the agent or its socket (such as a root user) can use it to authenticate, although the private keys themselves are not available directly from the agent. This makes agents (and agent forwarding) unsuitable for use on untrusted machines.

SSHKeychain

On the Mac, SSHKeychain is an excellent SSH agent. It can automatically load keys when needed, forgetting them when the system sleeps. SSHKeychain is accessible through the Dock or the SSHKeychain menu, and integrates with Apple's Keychain, optionally automatically loading and unloading SSH keys as the Apple Keychain unlocks and locks, and it works with other SSH-supporting programs such as BBEdit.

Once you've created and tested your private/public keypair, there are several simple steps to activate SSHKeychain.

  1. Open SSHKeychain.dmg to mount it.
  2. Install SSHKeychain (I just dragged SSHKeychain.app into /Applications).
  3. Unmount SSHKeychain.dmg.
  4. Open /Applications/SSHKeychain.app.
  5. Configure SSHKeychain's preferences (accessible from the new SSHKeychain menu on the right side of the menu bar); the critical parts are to enable "Manage (and modify) global environment variables", and configure specify your private keys if you used nonstandard filenames.
  6. Use Agent > Add All Keys from SSHKeychain's menu to confirm it can load your encrypted keys, and that now you can ssh into systems without entering a password. If you check "Add passphrase to the Apple keychain", SSHKeychain will no longer prompt you for the key's passphrase, although it may prompt to unlock your keychain to retrieve the key's passphrase.
  7. Set SSHKeychain to run automatically (I control-clicked on its Dock icon and set Open at Login).
  8. From now on, whenever you log into Mac OS X, SSH programs will use SSHKeychain for key management.

The SSHKeychain menu shows either of two icons. When it has one or more keys loaded, it shows a keyring with three keys. When it doesn't hold any keys, the ring is missing, but the keys are still shown.

There are a couple bugs in Mac OS X 10.4.x's authentication and the Apple Keychain. In theory, unlocking the screensaver should unlock the user's default keychain (if it uses the account's login password, which is Apple's default configuration), but this doesn't work properly. Additionally, while a keychain is unlocked, the system does not need to prompt the user to "unlock" that keychain again until it's re-locked, but this is also broken -- Mac OS X prompts to "unlock" keychains that are already unlocked.

In reality, if the system is running with the screensaver and Keychain locked, applications tend to bring up their own Keychain dialogs behind it. Unlocking the screensaver or the Keychain does not dismiss these dialogs. The upshot is that, with SSHKeychain set to lock the Apple Keychain on screen activation (its default behavior), unlocking the screensaver may reveal multiple Keychain dialog boxes which must each be addressed individually. With .Mac synching set to Auto, I often saw three authentication dialogs when I unlocked the screensaver.

As a partial workaround, open Keychain Access, create a new keychain (I called mine "lowsec"), select the new keychain, and use "Edit > Change Settings for Keychain "lowsec" to disable locking entirely for this keychain. Then go back to your main (login) keychain, and move the ".Mac password" item (its name will be your .Mac username) from the main login keychain into the new one. Additionally, enable "Use custom security settings" in SSHKeychain, and set "On screensaver:" to "No action". With this configuration, the lowsec keychain will be unlocked once at login time and then available until logout or reboot. For higher security I set my main login keychain to automatically lock after 30 minutes of inactivity. Ideally, both Keychain Access and SSHKeychain would allow locking specific keychains when the screen locks, leaving other keychains unlocked, but this is not currently implemented.

Key Restrictions and Forced Commands

It can be very useful to provide restricted access to an account. In SSH, this is possible by adding restrictions to specific keys in authorized_keys. Some of the possible restrictions include disabling interactive login; restricting agent, X11, or port forwarding (tunneling); and specifying a "forced command", which is always run when the associated key is used, regardless of what is requested by the ssh client. These restrictions are particularly useful for non-interactive tasks such as backup scripts. Such tasks may require SSH connectivity with unencrypted private keys, which should not provide unrestricted ssh access.

Note that multiple keys may provide access to the same account. This is handy for people sharing single accounts, or for using special-purpose keys with different forced commands. For more information on key restrictions and forced commands, see the "AUTHORIZED_KEYS FILE FORMAT" section of the sshd manual page.

Public keys can be further restricted by allowing access to carefully circumscribed accounts; unencrypted public keys which give access to "real" user or root accounts should be avoided as much as possible.

Raising the Bar: Requiring Stronger Security

There are several ways to protect a server against ssh-based attacks, including firewalls, TCP Wrapper, not creating or distributing UNIX passwords (forcing ssh public key authentication), and disabling password access for all accounts or the root account. To read about using firewalls and TCP Wrapper with ssh, please vote for Take Control of SSH.

UNIX passwords present several problems for administrators. What legitimate users can remember and type (generally considered to be 8 letters and numbers) is a small enough range of possibilities for attackers to try all possibilities. "Account lockout" is a feature of some systems (including Mac OS X Server) to disable accounts after several failed guesses -- which often identifies an attack. Unfortunately, this means legitimate users get blocked when their accounts are attacked, and locking legitimate users out of their own accounts is a successful attack (although not as serious as gaining illicit access). System administrators would often prefer to avoid this by not allowing password access at all. On Mac OS X, it is difficult to set up an account without a password; it's easier to create a long random password (12+ characters -- Keychain Access can do this for you), and never write it down or give it to the account user, requiring public key authentication or some other high security authentication (such as smart cards) instead. On other systems, it's easy to simply not set UNIX passwords for accounts (although there are complications).

OpenSSH has its own features to help force users to use public keys, blocking password guessing over the Internet. If you have enabled the root account (which is technically accomplished by setting a password for it), you should definitely set "PermitRootLogin without-password" in /etc/sshd_config; this will prevent people from breaking into the root account by guessing its password -- an amazingly common and disturbingly successful Internet attack. Even better, set "PasswordAuthentication no" to prevent sshd from accepting passwords for any account -- thus requiring keys for ssh access. For more about how to configure sshd and ssh, and enhance security with ssh, please vote for Take Control of SSH.

Take Control of SSH, Draft Excerpt: Reverse Tunnels

I've started writing something I hope will be an ebook called Take Control of SSH. It's an amazingly long process, so here's an excerpt -- the coolest trick, which initially made me want to write about SSH. This is for all those of you (us) who support family and friends remotely.

If you'd like to read the rest, please vote for Take Control of SSH.

Reverse Tunnel (Meet in the Middle)

This is a somewhat strange but very powerful feature. One of the problems with modern-day networking is that most computers are protected behind firewalls and/or Network Address Translation (NAT), meaning it may not be possible to connect directly to the computer you want to reach. For example, I sometimes want to remotely control my father's PowerBook to help with a problem, but he has a private dynamic address, behind an AirPort Extreme base station which itself has another dynamic address and provides NAT. Even if my father told me his IP address and his base station's address, I wouldn't be able to reach his PowerBook. Reverse tunneling works around this by allowing him to "project" a tunnel to another computer. If I can reach that computer, the tunnel connects back to his computer. Because he establishes the reverse tunnel from his computer as a client, he doesn't have trouble with (Network Address Translation) NAT, and he doesn't even have to know his own IP address. For reverse tunneling to work, my father and I both need access to a computer running sshd. In this example, my PowerBook is called pb, his is called dad, and the server is www.reppep.com.

Note that reverse tunneling provides access to the tunneling computer, and so could be a violation of network security policies. The tunnel is restricted (only accessible from the gateway computer, or other systems with access to it), but it is important to check with the network administrators before implementing a reverse tunnel. In addition, reverse tunneling is fairly complicated. A simpler option is to make SSH directly accessible, but unfortunately this often cannot be arranged. If you'd like to read more about configuring dynamic DNS and firewalls for direct SSH access, please vote for Take Control of SSH.

To use Remote Desktop through a reverse tunnel and an intermediate system, there are several steps (this can be simplified if the client is directly on the Internet and accessible via ssh). There might be as many as 3 separate encryption/decryption cycles in this scenario, but performance is generally decent.

These instructions were originally written for Apple Remote Desktop 2, but apply equally to Remote Desktop 3. Generally, I strongly suggest connecting with Remote Desktop 3's "Encrypt all network data (more secure)" option, but in this case it doesn't do quite what is needed, and would only add a redundant extra layer of encryption.

  1. On the computer to be controlled (dad in this example), make sure Apple Remote Desktop is enabled in System Preferences:Sharing:Services.
  2. On the computer to be controlled (dad), open a couple ports on the server and point them at the ssh and VNC servers on dad, by typing the following into a Terminal window. The sleep command ensures the tunnels will be accessible for at least an hour -- they stay open automatically while in use:
    ssh -R 6900:127.0.0.1:5900 -R 6922:127.0.0.1:22 www.reppep.com sleep 3600
  3. On the machine running Remote Desktop.app (pb in this example), set up tunnels to those ports on the server:
    ssh -C -L 6900:127.0.0.1:6900 -L 6922:127.0.0.1:6922 -o HostKeyAlias=dad www.reppep.com
  4. Now you can log into dad with:
    ssh -p 6922 www.reppep.com
  5. Alternatively, you can connect to 127.0.0.1:6900 with Remote Desktop (or port 6900/display 1000 in a VNC client), and control dad.

Note that if the firewall in Mac OS X (non-Server) is on, and any services are enabled in System Preferences:Sharing:Services, the appropriate network ports are automatically opened in the firewall, accessible to the entire Internet; this is overly exposed. OpenSSH can be restricted with TCP Wrapper; the Apple Remote Desktop agent (which is a VNC server at its core) cannot; VineServer has an option to only listen on the loopback address, so external hosts cannot connect -- or even detect the presence of a VNC server -- without first establishing an ssh tunnel.

http://www.redstonesoftware.com/products/vine/server/

To make this easier for my father, I made sure he uses the same login name on www.reppep.com as on dad, and I gave him a double-clickable file (ssh-tunnel.command) to run step #2 (the part he has to do each time I'm going to help him), in his Dock, to establish the ssh tunnel. Files ending with .command are automatically executed by Terminal when opened in Mac OS X.

To see what TCP ports are accepting connections, use "netstat -an | grep LIST". Step #2 creates tunnels on the server's ports 6900 & 6922 (connected back to dad), and step #3 opens the same port numbers on the client (pb), tunneled to the server.

When tunneling ssh connections through ssh, the ssh program tends to show a "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" alert, because ssh gets confused about which ssh server it's connecting to. The short-term fix is to edit ~/.ssh/known_hosts and remove the entry for 127.0.0.1 or localhost. If you would like to learn more about the cause and other options for dealing with it (including HostKeyAlias, shown in the example above), please vote for Take Control of SSH.

Thursday, March 15 2007

Locking ssh Access to Solaris Accounts

Today's tip is brought to you by the letters SUNW.

As we migrate from encrypted passwords to public keys for ssh access, one of the problems we have is how to make sure an account is disabled across many machines (we're not up to authenticating against a central LDAP or AD directory yet).

Given the variations in home directory location, eccentricities of escaping commands over ssh connections (dsh makes it even worse), difficulty in recognizing public key strings, and level of paranoia appropriate when closing an account, it's very difficult to be certain a person's access has been entirely cut off (especially if they had legitimate access to elevate privilege).

There are simply a lot of files in a lot of different places to check and edit, and no way to know you've found them all without spending some quality time with each host, when we want something quick and complete.

On Solaris (but not on RHEL4), if the password field in /etc/shadow is set to *LK*, the account is not accessible via OpenSSH and public keys, although it is via local su. Just to keep things interesting, OpenSSH prompts for a password, even though it knows the account is locked! If the password field is '*', '!', or a real encrypted password prefixed with '!' (a standard way of temporarily disabling a password), ssh access is permitted with a valid public key.

That doesn't guarantee the user didn't bury their public key in someone else's authorized_keys file, but doing this to other individuals would be clearly malicious and actionable, and doing it to the system accounts is more reasonable to check; better is to have a policy of simply replacing their authorized_keys files from known-good copies whenever there's a change.

Update 2007/05/11: In Mac OS X Server, you can uncheck "access account" in Workgroup Manager, but still log in with a public key.

Under OpenSSH on Solaris, authorized_keys (and the .ssh directory) can be owned by root, handy for centralized account management, but under RHEL4 they must be owned by the user.

Saturday, March 3 2007

The Apple Keychain is cool, but also strange and problematic

I'm writing about SSL/TLS on Mac OS X, so I've been playing around with certtool (a very nice text-based interactive tool for working with certificates, based around the Apple Keychain), and security (a more powerful tool for dealing with keychains in general).

I've discovered that not only can't certtool export the certificates (or keys) it generates (it only exports CSRs, which is important but insufficient), but "security export" doesn't work for me either. I can't tell if it's pilot error, but the documentation is sparse and inconsistent.

Needless to say, I left some work in Radar for Apple's doc folks and security programmers this week.

More confusing, when I generate a certificate with certtool it shows up as expected in Keychain Access, but I don't see the private key there. My understanding of certificates is that there must be a private key for every certificate (although one key might be used by several certificates), so where is it? Does every keychain have am embedded and invisible private key, which you never see? This has implications for people generating certificates, as they might not be able to disentangle and distribute them in the future, if multiple certificates are tied to the same secrete key. If I could get "security export" to work, I could see if it emits the key along with the cert, but so far no joy.

Just to make things more interesting(!), I have private and public keys in my keychains. They're not SSHKeychain items, which are identified as such (and really just each hold an encryption passphrase for a normal triple-DES encrypted OpenSSH private key).

So I suspect there is a 'bedrock' private key in each of my keychains, (probably automatically generated when the keychain is created), which I can't see or access directly. But I don't know! This would make sense, as the keychain items are all encrypted, so perhaps they are encrypted with the same private key the public keys are tied to. But keychain icons can be decrypted and re-encrypted with another key; certificates are permanently tied to their private key.

More bugs: When you open Keychain Access and Show Keychains, you select a keychain in the upper-left corner ("Keychains" list), and only see keys from that keychain. But if you type into the upper-right search box, Keychain Access switches to showing all matches across all keychains, while leaving one 'selected' in the upper-left -- except that selection no longer controls which keychain's items are displayed, so the meaning of the selection changes (goes away, actually).

But the cool part is this

Launch Keychain Access, and under its Keychain Access menu, select Certificate Assistant. The Certificate Assistant (actually a separate program) has explanations of private/public keys and certificates, and assistants for generating CSRs, self-signed keys, even your own CA -- all behind a helpful GUI. For bonus features, you can use it to review existing certificates, or even interrogate an HTTPS server for its cert dynamically and review the cert (similar to visiting the site in Safari and clicking the lock icon), and Apple provided some nice automation, so when you create a CA, it gives you a configuration file you can give to other people. If they open the file, the Certificate Assistant launches on the recipient's Mac, and walks you through sending a CSR back to the CA Mac. Crazy! It claims to have sent an email request for a certificate signing on my behalf, but I didn't confirm or see it happen, so I'm a bit confused (again). But it's very cool, no question.

Terminology

It's very difficult to be clear when talking about keychains. The Apple Keychain is an API, system feature, and file format (and apparently a framework, distinct from the Keychain Manager, SecurityFoundation.framework, and SecurityInterface.framework).

Each user and installation of Mac OS (including Classic) may have one or more keychains. Keychain Access is Apple's application for managing both the Apple Keychain (including overarching per-user settings) and individual keychains (both personal and system-wide). The Keychain MenuExtra offers a subset of the features in Keychain Access.

SSHKeychain is a third-party application that manages ssh agents, apparently derived from the Gentoo project's keychain (a popular command-line tool), but tied into the Apple Keychain, which may contain SSHKeychain items. Confused yet?

Even when I am able to keep it all straight, it's very difficult to explain this stuff to users who don't know all the pieces (anyone who already knows all the pieces probably doesn't need an explanation from me!), and what percentage of people do you think to get to the end of this terminology explanation?

Eating Your Own Dogfood

Apple has several obscure tools for dealing with certificates, including security, certtool, and the Certificate Assistant (and certadmin & Server Admin for Mac OS X Server). Reading their documentation, I keep finding obvious issues, which Apple should have found and fixed already.

I blame this on two main causes. First, each of these tools has a relatively small potential user base. While openssl is available on Macs, Linux, Solaris, and even Windows, certtool is only available for Mac OS X users, and only relevant for command-line users, whereas Certificate Assistant is only relevant for graphical Mac users (personally, I want command-line tools, which I can automate or use in a shell -- likely through an ssh connection). Second, these tools have a very specific primary audience -- it looks like many of them are to support Apple's own internal requirements. If Mac OS X Server needs certificates, someone at Apple writes a tool to build them. If they don't need export, certtool might not get an export option, even if that's core functionality for SSL users in general.

Secondary reasons include ties to proprietary Apple technologies (people who work with certificate files may not want, or be able, to deal with the Apple Keychain), and completeness. The manual page for security, for instance, describes it as incomplete (as certtool is), and some of these are only recently available.

The unfortunate result is that Apple provides a whole mess of almost-baked and almost unknown tools. Even Apple ignores their tools sometimes -- http://developer.apple.com/server/security_ssl.html uses CA.pl, included with OpenSSL, and barely mentions certtool, completely ignoring the Assistant.

Update

The missing private keys are a Keychain Access bug, and fortunately all your certs are created against independent keys by default, which is the right way to do it. Check out the comments for more (and thank you, Perry http://www.cynic.org/perry/).

Separately, I thought I was going crazy, trying to reconcile observed reality with http://developer.apple.com/server/security_ssl.html. The resolution turned out to be simple -- disbelieve! That document apparently describes Panther, and hasn't been updated since it become wrong. It largely discusses how to put a cert into a keychain file for the MOSX mail server, but as of 10.4, Apple's Cyrus installation doesn't use the keychain. I guess updating it for 10.4 would make the body of the article (about certtool and keychains) irrelevant, and nobody's bothered to replace it with something current.

pepper@www:~$ grep tls /etc/imapd.conf
tls_cert_file: /etc/certificates/mail.reppep.com.crt
tls_ca_file: /etc/certificates/ca.reppep.com.crt
tls_key_file: /etc/certificates/mail.reppep.com.key
tls_server_options: use
tls_common_name: mail.reppep.com
tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH

- page 1 of 2