Update 2009/07/02: Although I wrote this script specifically for comparing files, I have found that it also saves a lot of typing for editing individual remote files, cutting 14 characters of syntactic sugar down to 5: bbs<Tab> and a Space.

I often need to reconcile (partially or completely) one or more files on two or more machines. I often to use BBEdit's excellent "Find Differences" to do this, but loading the files into BBEdit can be awkward. I still hope someday Bare Bones will add support for remote files to their bbdiff command, but I'm not holding my breath. Today I whipped up a shell script to make it more convenient:

#!/bin/sh
# bsftp: Edit multiple files from multiple hosts.
# Usage: bsftp "host1 host2 host3" "/etc/passwd /etc/shadow /etc/group"

for file in $2
 do
  for host in $1
   do
    bbedit sftp://root@$host/$file
   done
 done

I've been fighting with Dell DRAC for months at this point. Some of their updaters insist that a copy of the updater is already running (even across reboots), and can only be installed by booting from a diagnostic CD. I've seen this with multiple updaters using DUP (Dell Update Packages).

The DRAC vKVM control doesn't work in 64-bit Firefox, and DRAC serves the ActiveX control to Firefox/Mac. When I point out this is a stupid bug, and the non-ActiveX plugin which works on Firefox/Linux should actually work on Macs, I'm told Macs are unsupported and this won't be fixed.

But today takes the cake. I wanted to install the new DRAC firmware on a PowerEdge 1950, so I downloaded Dell's DRAC5 Update Package for Red Hat Linux, v1.45, A00. I run RAC_FRMW_LX_R209365.BIN (well, of course that's v1.45 -- isn't it obvious??), but it requires an X11 DISPLAY. Why? It's not a graphical application. Dunno, just do it.

Okay, I log out, log back in with X11 forwarding, and run it. Dozens of prelink errors (about the same files, repeating) scroll off the screen, and then the xterm closes before I see what it did. Run it again, and it's failing because Dell expects a lockfile binary in my PATH. I don't have one. What does Google think? Workaround: yum install procmail, install, then rpm -e procmail to clean up. What possible excuse could Dell have for requiring (a piece of) procmail on my system to upgrade DRAC?

Then it failed again, but at least that time it spat out a useful error message:

Error while loading shared libraries: libstdc++.so.5:
cannot open shared object file: No such file or directory.
You must install the Linux compatibility libraries.
To install the compatibility libraries, use the following command:
"rpm -ih compat-libstdc++-33-3.2.3-47.3.i386.rpm"

Fortunately I was saved from banging my head against a wall any more -- I found Dell's "Hard-Drive" updater. That too is a Windows executable, but unzip f_drac5v145_A00.exe strips out the useful bit: firmimg.d5. The upload failed from Safari on my Mac, but I got it to work in Firefox/Linux (next time I'll try Firefox/Mac, which should work).

Parting Peeves

Why does Update sit under the "Remote Access" subhead, rather than the "System" heading? It's firmware for the whole DRAC system, not just the Remote Access part. I shouldn't have to hunt for the upgrade page. Adding insult to injury, the (remote) Console & Media tabs are under System, not Remote Access. Does anybody think that makes sense?

In a final bit of brilliance, as the updater reached 100%, I got a warning that my session was about to be closed. Next time, guys, make sure you don't break your own firmware updates by timing the user out in the middle, okay?

Sun's How to Manually Update the Boot Archive on a RAID-1 (Mirror) Volume procedure says to find the root slices from the console messages -- which note md devices discovered during boot -- and embed these into /etc/vfstab to fix the boot environment and temporarily disable mirroring. Unfortunately, this guidance is incomplete and incorrect.

The first problem is that Sun's documentation instructs you to boot from 'the primary submirror.' But of course it might be corrupt (something scrambled the boot archive, after all). This week, one of our submirrors for / and both submirrors for /var showed errors under fsck -n. c4t0d0s0 (the default boot device) had problems which prevented the system making it all the way to normal multiuser state. c4t0d0s3 had moderate corruption, while c4t4d0s0 had minor corruption. Fixing /var/ was tricky, because Sun does not have any documentation which I could find on how to recover good data from one submirror onto one with bad data. They assume the only failure mode is a dead disk, and disk replacement is simple. The undocumented trick is that 'Submirror 0' is authoritative for resync operations.

root@jean:/# metastat d3|head -9
d3: Mirror
    Submirror 0: d23
      State: Okay         
    Submirror 1: d13
      State: Okay         
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 8385930 blocks (4.0 GB)

Problem #2: fixing / is harder, because the procedure for booting from a different disk is totally obscure and non-standard. Basically, you must edit /boot/solaris/bootenv.rc, which overrides /boot/grub/menu.lst. I don't know why Sun apparently created a brand-new findroot command for grub, but doesn't actually run Solaris from the disk it specifies. At a guess, it stems from Sun's dissatisfaction with the way Linux & GRUB deal with the horrible multi-stage boot procedure required on x86 PCs. bootadm(1M) says it updates the 'boot archive', but not what a boot archive actuall is, and that it also updates the GRUB configuration, but our menu.lst hasn't actually been updated since I installed the system.

root@jean:/# grep bootpath /boot/solaris/bootenv.rc 
setprop bootpath /pci@1,0/pci1022,7458@4/pci11ab,11ab@1/disk@0,0:a
root@jean:/# grep -v \# /boot/grub/menu.lst 
default 0
timeout 10
splashimage /boot/grub/splash.xpm.gz
title Solaris 10 10/08 s10x_u6wos_07b X86
findroot (rootfs0,0,a)
kernel /platform/i86pc/multiboot
module /platform/i86pc/boot_archive
title Solaris failsafe
findroot (rootfs0,0,a)
kernel /boot/multiboot kernel/unix -s
module /boot/x86.miniroot-safe

If you find yourself in single-user mode with a root device like /pci@1,0/pci1022,7458@4/pci11ab,11ab@1/disk@0,0:a, rather something more normal like /dev/md/dsk/d0 or /dev/dsk/c4t0d0s0, it probably means Solaris is running from a device which it cannot correlate back to a valid boot device, although you can do this manually by examining the slice symlinks in /dev/dsk/:

root@jean:/# ls -l /dev/dsk/c0t4d0s0 
lrwxrwxrwx   1 root     root          62 Dec 30 14:13 /dev/dsk/c0t4d0s0 -> ../..
/devices/pci@0,0/pci1022,7458@1/pci11ab,11ab@1/disk@4,0:a

The final major problem is that disk device paths are not stable on the X4500. Sun's instructions are to find the disk path to the root submirror from console messages (in my case, they referred to /dev/dsk/c3t0d0s0 & /dev/dsk/c3t0d0s0) and use one of these in /etc/vfstab rather than the metadevice, but when I actually booted into Solaris, those slices didn't exist, because the bootable disks were at c4 rather than c3. I had to boot back into GRUB's Failsafe mode and correct the device for / in vfstab. Sun's documentation is fine for machines with consistent disk paths, but wrong for the X4500 (and presumably the X4540 as well).

Additionally, I worried that bootadm might read c4 from the vfstab file and write to c3 (a ZFS pool disk it must not modify!), or something similarly screwy, but this turned out to be a non-issue once I'd sorted out the rest.

Tip: bootadm apparently normally caches its changes and writes them to disk when rebooting; to force bootadm to write changes immediately, add the undocumented -f flag, e.g.: bootadm update-archive -fR /a

bootadm(1M), of course, doesn't provide any useful detail on what it does.

The Problem: Ethernet >>> AirPort

For whatever reason, my MacBook Pro doesn't get very good network performance over 802.11n AirPort. Since I routinely copy everything from half-hour MPEG videos (hundreds of megabytes) through full DVDs (several gigabytes) between it and my Linux server, I much prefer gigabit Ethernet.

It turns out Apple's AFP & SMB clients are smart enough to seamlessly migrate a network connection from one transport to another -- if you disconnect from Ethernet but AirPort remains up, Mac OS X will reconnect to the file server via AirPort. Contrawise, if Ethernet is connected when AirPort goes offline, the connection will switch back. Note that I haven't tested with Ethernet & AirPort in different subnets.

This means once I have reconnected the Ethernet cable, I have 2 ways to switch my SMB connection back: I can unmount all shares from that server and then remount, or I can bring down the AirPort connection and force the OS to migrate my connections over. I prefer not to bring down AirPort, because that breaks any other open AirPort connections, which might not reconnect (iChat reconnects; Safari downloads just fail), but I do this sometimes when a program is using the share and preventing the umount from succeeding.

For over a year, I've been using a shell alias to handle the disconnect/reconnect -- below is the final version. The grep was so I could confirm it worked -- if I see prowlere (Ethernet interface), I know I have a fast connection. If I see prowler instead, I still have a slow AirPort connection -- most likely because an open file prevented the umount.

alias remount='umount /Volumes/inspector; umount /Volumes/dvd; umount /Volumes/1tb; open smb://pepper@inspectore.rep.dom/inspector/; open /Volumes/inspector/home/pepper/tivo/; sleep 10; netstat -a | grep \.micro | grep ESTABLISHED'

The Improvement: Automation

Back in the Jaguar era, I had a script to run Plucker automatically. I had my laptop configured to run it both via cron and whenever I (dis)connected my Ethernet cable, but when the network trigger broke I just refined my crontab to compensate and forgot about it. In Leopard, launchd handles tasks like this, but it doesn't offer network triggers.

I wanted to recreate the automatic trigger on network reconnection, but wasn't sure how to do it. crankd could probably do the trick, but I don't know Python. Fortunately Jeremy Reichman pointed me to MarcoPolo, which fits the bill admirably.

I was initially confused by the fact that MarcoPolo automatically copied the system's Network 'Locations', which I don't want to change. Fortunately MarcoPolo is happy to work with its own 'Contexts', and leave the system Location untouched. I created a couple contexts, 'en0 online' and 'en0 offline', told MarcoPolo to use IP and NetworkLink as "Evidence Sources", and configured it to switch to 'en0 online' when the 'prowlere' IP comes online. On the other hand, "en0 (Ethernet) link inactive" switches to 'en0 offline'. This way if I connect to an outside network and get a different Ethernet IP, it won't try to connect to my home server. MarcoPolo can run shell scripts, so I converted my alias to a script, which runs on switching to 'en0 online':

pepper@prowler:~$ cat bin/remount 
#!/bin/sh
# remount: Reconnect to inspectore, hopefully via Ethernet rather than AirPort

umount /Volumes/inspector
umount /Volumes/dvd
umount /Volumes/1tb
open smb://pepper@inspectore.rep.dom/inspector/
sleep 10
netstat -a | grep micro | grep ESTABLISHED | awk '{print $4, $5}' | growlnotify -w

Future possibilities include launching iPhoto when I connect a camera memory card but not when I plug in the iPhone (I already had 37signals' workaround, but MarcoPolo would have been a simpler option), or switching iTunes to the upstairs speakers when I connect to the upstairs monitor (iTunes' output device is not currently scriptable, so I filed an RFE). Unfortunately, since these triggers and actions are all orthogonal to each other, I'd end up multiplying contexts to accommodate them, which is suboptimal.

David, thanks for MarcoPolo!

I just spent a while learning (the hard way) about Cisco Port Security, so here's what I got out of it.

Cisco Port Security, when enabled, keeps a whitelist of allowed MAC addresses for each port; the list may hold 1 or more entries, and might be statically specified by an administrator or automatically 'learned' by the switch.

Intuitively, you might expect something called "port security" to prevent unapproved hosts from sniffing traffic, as this is the most serious risk. The reality, however, is that the switch has no way of knowing who's listening to a port, only of knowing who's transmitting individual packets, because each packet is 'signed' with the transmitter's MAC address. With PS enabled, the switch silently drops packets from unapproved MACs, preventing them from stealing bandwidth or actively attacking the network. These are useful, but data and credential sniffing are more serious risks, and are not addressed.

Complications

1. Front-line support may have limited knowledge of security protocols and mechanisms, and is unlikely to have access to directly check the list of banned MACs/ports. Cisco provides a mechanism for alerting, but that does not mean notifications have been configured to reach all the concerned parties.
2. Sniffing to determine the situation will show the host generating outbound traffic, but will not reflect that these packets are not being forwarded to any other hosts.
3. Depending on situation, even after PS activates, there may be residual traffic for the blocked device. This can look like responses to current traffic, and mask the fact that the muzzled node is in fact mute.
4. Using DHCP seems to avoid some of this. I found that using DHCP got an address and enabled full communications. I don't know why.

As Murphy would have it, I had just opened up the server chassis and enabled ipf before PS tripped, so I spent a lot of time looking for a (missing) misconfiguration.

See Also

• http://www.cisco.com/en/US/docs/switches/lan/catalyst2950/software/release/12.1_6_ea2c/configuration/guide/swgports.html#wp1043841
• http://www.linglom.com/2009/01/20/secure-the-network-using-cisco-port-security-part-i-concept/
• http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.1E/native/configuration/guide/port_sec.html
• https://www.conetrix.com/Blog/post/Cisco-Port-Security-and-Sticky-MAC-Addresses.aspx

Amy and I watch video on our Apple TV, and I watch on my iPhone. I rip to filenames ending in .AppleTV.mp4 (for us to watch together) and .iPhone.mp4.

Under iPhone: Videos: Sync movies, iTunes 8.1 now greys out incompatible videos. This is a significant improvement, and makes selecting videos for the iPhone easier & quicker.

Thanks, Apple.

Update 2009/03/10: It looks like mke2fs is smart enough to automatically select the 4k blocksize, and largefiles4 is not necessary (which is good, as it was interfering with our backups).

We compared performance between 10-disk and 20-disk RAID6 sets on a SATABeast, and discovered the performance difference is not significant, so we chose the most efficient reasonable layout: 2 20-disk RAID6 sets, each containing a single volume the same size. These appear to the Linux host as a couple 16.37tibyte LUNs. We're using device mapper multipathing to provide fault tolerance across both FC paths (in Nexsan's recommended "All Paths All LUNs" mode, each LUN is available via both controllers). This is all handled (except the performance testing) via the SATABeast administration interfaces.

Within Linux, we create 2 LVM logical volumes of just under 8tibyte (the largest ext3 can handle), and a third with the leftover 384gibyte, from each LUN.

The SATABeast lets the host see and use the volumes while it's still generating parity on the underlying RAID arrays ("Online Creation"), but creating file systems is much slower during this process.

A fully configured SATABeast contains 42 1,000,137,687,040-byte ("1 terabyte") drives. They reserve 2 for spares, so we have 40 disks to work with. Nexsan suggests 4 10-disk RAID sets, but RAID 6 allocates 2 disks per RAID set to parity, so with 4 10-disk sets we would 'waste' 10 disks, and our usable space would be 4 volumes, each 8 usable disks = 8,001,101,496,320 bytes / 1024^4 * 4 RAID sets = 29tibyte usable. 29tibyte is a lot, but only 70% of the specified "42tbyte", so we'd really like to be more space efficient -- we have 2 hot spare and 4 p

I set up /etc/fstab earlier, using ext3 labels.

How to create an ~~8tibyte ext3 filesystem on a large multipath raw volume.

1. pvcreate /dev/mapper/mpath3
2. vgcreate satabeast1vg /dev/mapper/mpath3
3. vgdisplay # Note number of usable extents (PE).
4. lvcreate -n satabeast1lv --extents 2096128 satabeast1vg # Use number from above. # It least close to the largest valid ext3 volume size.
5. mkfs.ext3 -L/satabeast1a -Tlargefile4 -b4096 /dev/satabeast1vg/satabeast1a # If the SATABeast is still calculating parity, this takes a while. Go get some food...
6. vi /etc/fstab ## I use something like LABEL=/satabeast1 /satabeast1 ext3 defaults 0 0
7. mount /dev/satabeast1vg/satabeast1lv /satabeast1
8. df -h /satabeast1

Here's my transcript of creating an LVM volume with two not-quite-8tibyte and one 348gb filesystems on a 16.37tibyte LUN.

[root@norimaki device-mapper-multipath-0.4.7]# pvcreate /dev/mapper/mpath3 
  Physical volume "/dev/mapper/mpath3" successfully created
[root@norimaki device-mapper-multipath-0.4.7]# vgcreate noribeast0vg /dev/mapper/mpath3 
  Volume group "noribeast0vg" successfully created
[root@norimaki device-mapper-multipath-0.4.7]# lvcreate -n noribeast0lv --extents 2096128 noribeast0vg
  Logical volume "noribeast0lv" created
  Logical volume "noribeast0lv" successfully removed
[root@norimaki device-mapper-multipath-0.4.7]# lvcreate -n noribeast0a --extents 2096128 noribeast0vg
  Logical volume "noribeast0a" created
[root@norimaki device-mapper-multipath-0.4.7]# lvcreate -n noribeast0b --extents 2096128 noribeast0vg
  Logical volume "noribeast0b" created
[root@norimaki device-mapper-multipath-0.4.7]# vgdisplay
  --- Volume group ---
  VG Name               noribeast0vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               16.37 TB
  PE Size               4.00 MB
  Total PE              4292125
  Alloc PE / Size       4192256 / 15.99 TB
  Free  PE / Size       99869 / 390.11 GB
  VG UUID               QLdntg-9ccY-0HYe-DnWI-Lxxu-Huzj-S3bExH

[root@norimaki device-mapper-multipath-0.4.7]# lvcreate -n noribeast0c --extents 99869 noribeast0vg
  Logical volume "noribeast0c" created
[root@norimaki device-mapper-multipath-0.4.7]# mkfs.ext3 -L/noribeast0a -Tlargefile -b4096 /dev/noribeast0vg/noribeast0a
mke2fs 1.39 (29-May-2006)
Filesystem label=/noribeast0a
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
8384512 inodes, 2146435072 blocks
107321753 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
65504 block groups
32768 blocks per group, 32768 fragments per group
128 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@norimaki device-mapper-multipath-0.4.7]# time mkfs.ext3 -L/noribeast0b -Tlargefile -b4096 /dev/noribeast0vg/noribeast0b
mke2fs 1.39 (29-May-2006)
Filesystem label=/noribeast0b
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
8384512 inodes, 2146435072 blocks
107321753 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
65504 block groups
32768 blocks per group, 32768 fragments per group
128 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544, 1934917632

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

real    9m52.451s
user    0m0.657s
sys 0m3.660s
[root@norimaki device-mapper-multipath-0.4.7]# time mkfs.ext3 -L/noribeast0c /dev/noribeast0vg/noribeast0c 
mke2fs 1.39 (29-May-2006)
Filesystem label=/noribeast0c
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
51134464 inodes, 102265856 blocks
5113292 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3121 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 27 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

real    1m45.683s
user    0m0.172s
sys 0m10.150s
[root@norimaki device-mapper-multipath-0.4.7]# df -h |grep nori
/dev/mapper/noribeast0vg-noribeast0a
                      8.0T  175M  7.6T   1% /noribeast0a
/dev/mapper/noribeast0vg-noribeast0b
                      8.0T  175M  7.6T   1% /noribeast0b
/dev/mapper/noribeast0vg-noribeast0c
                      384G  195M  365G   1% /noribeast0c

James and I toured Comic Con today. James enjoyed being there as a civilian, rather than a reporter/interviewer or a booth bimbo, and I kept noticing how much more interesting-looking the crowd was than a Macworld Expo, PC Expo, or Linux Expo crowd.

I bought Ultimate Spider-Man 1 (to read to Julia, although perhaps not yet) and The Graveyard Book, which I had to read after reading Gaiman's award post. I got both from the Comc Book Legal Defense Fund, where Sharon & James both helped out.

Additionally, I got some sketches for James, Sharon, Julia, & myself (Iron Man for me; Iron Man in color for Julia). I got an R/C Supreme Dalek for Amy & myself. Amy was pleasantly delighted (as was Julia).

For a bonus treat, I saw Kirsten Cappy, whom I hadn't seen since Wheaton. From her, I bought the first two Capt'n Eli books, which Julia should be ready for.

I even got a few pictures.

Now I'm off to read Ultimate Spider-Man 1 to see if Julia's ready for it. The work of a Dad is never done. ;)

I was listing out suggested iPhone/touch apps for a friend (several people have asked for recommendations), and decided to post my current list of suggestions. Note that the Lite apps are also available in Pro versions -- I purchased several after liking the Lite versions. Numbered items are roughly in order of importance. There are several apps I downloaded them because they are interesting, even if I haven't really used them yet. All except the last version are available in free versions, although I have paid to upgrade 3 of my 6 top items.

See also my 2008/08 list.

Highly Recommended

1. NetNewsWire
2. Twitteriffic (Lite)
3. Instapaper (Lite)
4. Remote
5. Wikipanion (Lite)
6. Shazam

Mildly Recommended

1. 1Password (requires 1Password/Mac)
2. Facebook
3. Amazon Mobile
4. Showtimes
5. Lightsaber Unleashed
6. Google Earth
7. Yelp
8. Stanza
9. Darkslide
10. iHandy Level Free
11. Rain Stick
12. Scribble
13. Shakespeare
14. White Noise

Games

• Space DEADBEEF
• Labyrinth Lite
• Sol Free
• Tap Tap Revenge
• Crossword Light

Interesting

• AroundMe
• iWant
• iTalk Recorder
• Joost

Worth purchasing.

• Classics
• Toy Bot Diaries (1-3)