Extra Pepperoni

The Pepper clan flew to Martha’s Vineyard Thursday (Julia gets carsick, so we couldn’t drive that far). We arrived at LaGuardia an hour early, sailed through check-in, and found ourselves at the gate wondering where our plane was — late. It showed up, perhaps 25 minutes late, and they announced that now we would wait for the flight crew. What? It’s not like the crew were 5 minutes late. Assuming it takes a normally new flight crew just 10 minutes to take over the plane, they should have been there 35 minutes earlier if the plane was on time. That’s adding insult to injury, although fortunately we weren’t connecting to anywhere (from Martha’s Vineyard? Where?). Then the plane sat on the runway for a while, which is just normal screwed-by-airlines SOP.

Coming back today, though, was much worse. Our flight was supposed to take off from Martha’s Vineyard Airport at 12:18pm. We got there by 11:30, and checked in. I had “reconfirmed” on the phone both times, but it was useless as you can’t go through the TSA screening without actual boarding passes, which you can’t get by “reconfirming” on the phone. I tried doing it on their website, but lost the cellular connection before I could perhaps print the boarding passes (cellular service on Martha’s Vineyard would be a joke, except there’s nothing funny about it). Presumably this might help in some unusual case, such as if we were late arriving for an overbooked flight and they were deciding who to boot, so we won’t count wasting my time with reconfirmation as a screw-up, but while waiting in line to get a boarding pass it felt like one.

The flight normally takes about 50 minutes, so the plane should have been away from the gate and taking off when we were checking in. I went back a few minutes later to ask a question, and was told that the plane actually had not taken off yet, due to a serious mechanical problem (which the rep thought had since been fixed), but was expected to do so soon. We told the other people waiting for the flight, who universally asked “Why didn’t they announce something?!?” Apparently the USAir personnel decided not to announce anything until they heard that the plane had taken off, so they could provide an ETA. This was clearly the wrong decision, as we were all angry to be kept in ignorance — nobody had any idea there was a delay until I passed the word. I had verified the flight was supposedly on-time earlier that morning. The rep said, at least twice, that they would have cancelled the flight from LaGuardia completely, except that there were so many people waiting to take it back from the Vineyard. This of course made us wonder if they were about to cancel it entirely.

At this point it was 12:45, and we had been there over an hour. We didn’t think the plane could possibly take off, fly from LaGuardia, unload, and be ready to board before 1pm. We were all aggravated and bored. I asked the rep when the flight might board, and he initially said 1pm, then hedged by suggesting that if we left, we should be back by 1:45. Rather than sit in the airport for another hour, we went to the Martha’s Vineyard Glass Works, and watched them start making a glass sculpture.

At 1:27, I called USAir to ask if the plane had left LaGuardia. The rep told me that the flight was expected from MVY to board at 1:50. I asked if it had taken of, and she told me it was expected to board at 1:50. I said she’d already told me that, but had it taken off yet? She told me it was expected to board from MVY at 1:50, and land at 3:01. I explained that she had already told me this three times, and I wasn’t asking that question. She got it, and put me on hold for 10 minutes. When she picked up again, she apologized, told me she couldn’t give me any more information (#8), and told me the 1:50pm projected boarding time was the best she could provide. This call took 14 minutes and 1 second according to my Treo, so the 4th time she told me “1:50pm” was at 1:41.

We drove back to the airport (arriving at 1:47, again per my Treo), and noticed it was emptier than when we left. I asked the woman behind the counter what was up (thinking the flight had been cancelled and everybody had left), and was told it had left “10 minutes ago”, so the plane must have landed by 1:30 at the absolute latest, and been fully boarded and taxiing for take-off at 1:41, when the phone rep was telling me it the flight would be boarding at 1:50!

Let’s recap (#1-3 were Thursday; #4+ were Monday):

1. The plane to MVY was late getting into NYC; we don’t know why.
2. The crew was even later getting to the plane.
3. The plane sat on the runway — I don’t know if this was happening to everybody at LaGuardia, or we missed a window.
4. The plane from NYC was late leaving (unspecified “major mechanical fault” — that’s reassuring).
5. No announcement.
6. The airport rep told us to be back by 2:00, then 1:45.
7. The phone rep was completely unable to find out if our flight had even taken off. If she’d gotten this information, we would have scooted back to the airport immediately.
8. The phone rep kept telling me our flight would board as late as 1:41, after it had already boarded. If she had given me correct information when I called at 1:27, we apparently could have made the flight.
9. When the plane arrived unexpectedly, they loaded up everybody they could find, but not everybody. They did not wait for us, even though we had picked up boarding passes at the airport, I’d spoken with the rep behind the counter at least 4 times, and he had told us to come back at 1:45. Being late is annoying, but leaving early is great if all your passengers are on-board. If you know you have missing passengers, and you have an opportunity to beat the (revised) schedule, don’t take it. The plane we eventually got was a Saab 340B, with 34 passenger capacity. If the 12:18 was the same model and full, the missing three of us were 9% of the passenger complement. They should have waited the 10-20 minutes, to the announced time, before leaving. Note that this is a tiny airport. They have two “Gates”, which means two planes can be going through the TSA check-in at once, but everybody walks through the same chain-link fence, out onto the runway, to board the planes. 34 passengers is a large plane for this field. We arrived on a 19-passenger plane, and before we left, we watched a Cape Air flight which completely filled up with a single family. Getting us onto the plane once we showed up would not have been a logistical nightmare, as it could at LaGuardia.
10. The airlines love to collect personal information and give it to the US Government. But USAir did not call my cellphone to tell us the plane landed ahead of expectations; they didn’t even call our home number, which would have been useless but shown that someone was making an effort. Nobody was.

In retrospect, it seems most of this is the fault of “USAir Express, operated by Colgan Air” personnel in LaGuardia. They were late getting the plane and crew to the gate (we of course have no idea why). They were late leaving (we don’t know the nature of the “major mechanical fault”, or if it could/should have been caught earlier, or if the LaGuardia crew were in fact fixing a problem they found, which flew in on the previous flight from MVY, but it was in their laps when it left late). They didn’t tell the MVY or phone agents what was going on, or even when the flight left. Had this information made it to the phone reps or automated computer system which exists to track this sort of thing, we would have made the (delayed) flight.

The MVY people found a plane at their airport, dumped everybody they could find onto it, and pitched it back to LaGuardia. The woman who rescheduled us onto the next flight (which left at 3:40) was visibly embarrassed at the fiasco, but by the time we spoke to her we had already missed the flight. She said they never even knew the plane was in the air, until it landed. That’s beyond poor communications.

Congratulations, USAir (Express, operated by Colgan Airlines) — I am impressed — you found more ways to screw up than I thought possible.

Eric Warnke has discovered that, if asked nicely, SSHKeychain will print out the passphrase used to encrypt a loaded private key. This is bad, as the whole point of an ssh agent/keychain is to provide secure access to encrypted keys, meaning you cannot get the passphrases or plaintext keys out.

http://www.sshkeychain.org/pipermail/users/2007-August/000098.html

Crud on a cracker, Batman!

Verified in the latest (v0.8.1) — hopefully there will be a patch soon, but this just shouldn’t be possible.

http://www.sshkeychain.org/

I’ve spent the past 90 minutes or so scanning over 100 CDs into Delicious Library. These aren’t even CDs I particularly want to keep — I’d be happy to lala them, but it’s great to have such a simple way to build a catalog.

I’m wishing for better integration already. DL doesn’t pick up albums or album covers from iTunes, even though obviously there’s a very strong correlation between CDs I own and complete albums in iTunes. Further, it doesn’t interface to lala — they already have a list of CDs I own, and automatic updating between lala and DL as I buy, send, and receive CDs would be excellent. Likewise, building web pages from the catalog requires a third-party utility (fortunately free).

This might be helpful for my parents, who own a 300 CD changer and have a lot of trouble keeping track of what’s in it — but I haven’t gotten that far yet.

DL does books, movies, music, and games. It works most easily by scanning UPC barcodes with an iSight — I’d heard about it and thought it sounded great, but only recently gotten a computer with an iSight built in. As it turns out I could probably have used my Sony TRV25 DV video camera, but this is fine. I expect to have all our CDs and DVDs within a month; afterwards I’ll start scanning books. Not sure if I’ll ever use the database for anything important, but it will be excellent to have a database.

For friends like James and Matthew who have out-of-control CD collections, this could be a huge deal.

DL serves as a good illustration of the differences between physical and virtual. Holding the CD in front of the camera is annoying and slow, while sucking information down from the Internet is fast. I suspect part of this is false perception, though, as it may well be pinging Amazon with tentative bar-code scans until one is verified, meaning the Internet lookup is already halfway done, and contributing to what feels like scanning time.

I only had one bar-code that read wrong consistently, and one CD (Snapshot: Live At the Iron Horse, by Livingston Taylor) where if I put the artist in DL doesn’t know what it is — on the other hand, if I leave the artist blank, DL correctly picks up the Amazon profile — which shows Livingston Taylor on the site. Very strange.

Tip: Scanning got much faster after I rotated my MBP slightly. It was about 2′ away from the wall, and once I turned the back of it slightly towards a ceiling light, so there was better illumination on CDs in front of the iSight, scanning took less than half as long as it had been. This is mentioned in the help, but I had thought there was sufficient light because it worked. I was impressed with the difference a small rotation made — perhaps Delicious Monster will add a low light warning in a future version.

Unfortunately, Amazon doesn’t format titles the way I do, so I’ll consider DL a reference, and iTunes the master.

Today Paul (a neighbor we met at the playground water fountain last week) called the fire department because he saw black smoke billowing out of a building across 2nd St (again within a block of us and visible from our apartment, but a bit farther away this time). They sent 5 trucks, found a burning boiler producing CO, and put it out. Julia and I got there as they were about to pull out.

It sounds like he called in time, and everybody’s fine. Here’s hoping there are no more FDNY emergencies in this area for a while.

I passed the RHCE exam, hooray!

This was much easier than becoming Dr. Pepper, and much easier & safer than becoming Sgt. Pepper.

I spent Monday to Friday this week in RH300, the Red Hat Certified Engineer Rapid Track Course; today (Friday) was the exam. I believe I passed — they should email my final results by Wednesday. In reality, I took the test as much for the RHEL5 update as for the certification.

I was concerned about problem solving with no Internet access, no access to another system (in real life we almost always have another live system to check things out, as opposed to troubleshooting grub on an unbootable system without working man, which was a problem in class), and no ability to discuss with co-workers, but it’s not an exam about how well you can find answers on The Google, so this was the only realistic way to do it.

My bug count with Apple has been climbing lately, as I’ve been playing with Leopard betas a lot. At the moment, I have at over 1,100 bugs in Apple’s Radar bug tracking system. I was shooting for 2,000 by Leopard’s release (scheduled for October 2007), but I don’t think I’m going to make it. The actual (web-based) bug reporting process is just too time-consuming. I haven’t even gotten to test the good bits yet!

I’ve decided to adjust my aim a bit. I hope to have 1,500 bugs in Radar by Leopard’s release, and 2,000 by January 1, 2008. It’s good to have goals.

Update: I hit 1,205 on 2007/08/14.

We noticed several fire engines and a crowd of people standing on 5th Ave and 2nd St, looking at a building across the street (on the west side of 5th Ave, between 2nd & 3rd Sts). We initially thought Gary’s building was on fire, but there was no smoke — just a couple cherry pickers working at an empty building in the middle of the block.

Ironically, we noticed on Sunday that you can see right through that building, which we hadn’t really noticed before, and I took some pictures:

Very strange, and we still don’t know what happened. We hope nobody was hurt.

http://www.reppep.com/~pepper/album/353-2nd/album/353-2nd/5th-ave-fire/

Update 2007/08/10: They have boarded up the windows, and continued building on top. It doesn’t look like there was a fire; Scott & Christine think there was some sort of collapse of the construction work they’re doing.

The Register has an interesting article on how to “safely” take your computer to Defcon, with the very wise suggestion that you’re safer if your laptop does not go to Defcon. Cellular phones without 802.11 are probably okay for this year at least. They refer back to a much more hard-core SANS post on the same topic.

The exercise is more involved for the fully paranoid, or generally when preparing to enter a truly hostile network. I assume that someone at Black Hat/Defcon has an unannounced exploit that I’d be vulnerable to. This implies you shouldn’t have any sensitive data or access to sensitive machines. Since you wouldn’t need a laptop without data or access, you probably need to mitigate the consequences of getting hacked.

1. Make up a couple disposable passwords just to use at the conference, one for this machine and one for outside accounts. Destroy them later.
2. Bring an empty USB thumb drive.
3. Create a new email account, so you can send yourself notes/presentations for later.
4. Forward your important email accounts to the new one (keep copies on the normal accounts), so you don’t have to check them.
5. Note that if you have a hosting plan like DreamHost’s, you can create brand-new ssh and email accounts free. I believe DH offers SSL webmail, if you can ignore the certificate warnings.
6. Get a cheap monthly VPN account, as suggested by Glenn Fleishmann; this is much simpler than establishing a trusted Squid proxy on a non-sensitive box, as originally suggested by SANS; note that you are then trusting the VPN provider.
7. If you have any untrusted protocols, try to access them from your temporary shell account via ssh, or though an ssh tunnel.
8. Back up your data (image the drive — you will need it later, and a full image is fastest to restore).
9. Wipe your laptop.
10. Install your OS, creating a new account with your new local password.
11. If you have a built-in webcam with an independently software-controlled active light, tape over it. If you feel comfortable opening up your laptop, disconnect its internal microphone.
12. Create a new ssh keypair. If you know the netblock, only allow access from Defcon machines and your own personal host(s); I have some info on doing this in authorized_keys in Take Control of SSH, Draft Excerpt: Public Key Authentication. Make sure your laptop trusts your other (home) machines.
13. Only as needed, trust this new key on your other systems.

14. In your local firewall, block outbound access except to the ports you intend to use; this is easy in Linux, but a bit more complicated on Macs, where you need to write your own startup script (or .command script in Login Items). This is obviously overridable, but an effective way to make sure you don’t accidentally connect without encryption, either from habit or because a website redirects you to unencrypted HTTP to save encryption cycles (common). For services where you know what host you’ll be connecting to, embed that. Here’s a sample of what you might add to add Apple’s ipfw. Note that it’s easy to shoot your own foot off with outbound firewall restrictions.

    ipfw add 01010 allow tcp from any to 4.2.2.1 dst-port 53 out
    ipfw add 01020 allow udp from any to 4.2.2.1 dst-port 53 out
    ipfw add 01030 allow tcp from any to 4.2.2.2 dst-port 53 out
    ipfw add 01040 allow udp from any to 4.2.2.2 dst-port 53 out
    ipfw add 01110 allow tcp from any to any dst-port 22 out
    ipfw add 01120 allow tcp from any to any dst-port 443 out
    ipfw add 01030 allow tcp from any to smtp.dreamhost.com dst-port 587 out
    ipfw add 01040 allow tcp from any to mail.dreamhost.com dst-port 993 out
    ipfw add 01900 deny tcp from any to any out
    

15. Restore only required information to your laptop.

Enjoy the conference. Hi, Rich!

When you’re back home, connect from your home machines to the untrusted laptop, rather than the other way around, retrieve any data on it, and then boot from CD/DVD/PXE and reinstall, or restore from your image if you can do that without using the untrusted OS on the laptop’s hard drive.

The Super-Tent/IT Pavilion/Big Top/Big House fronts on the main RU parking lot, at the other end of which is the 66th St Gate. Except that after we moved in, they walled in the lot and started digging:

They still haven’t started on Smith Hall, though, which makes me wonder why we couldn’t still be in a proper building now. In the meantime, the main campus entrance and driveway are closed, along with the parking lot, under which a new electrical vault will be built. Getting around campus is much more complicated now than 6 months ago. This is especially true for IT, moving equipment around the tent, as the pathways and steps around the periphery don’t quite work for carts.

Our new main data center is nearing completion. It was previously our backup/disaster recovery site, so needed a lot of build-out to fit the rest of our servers. The swap from the older/smaller UPS system to the newer/larger one will be tricky, as several live servers will be switched over while running. Later we get to swap systems end-for-end across campus, so the primaries are in the primary DC, once their current location becomes the DR site. Needless to say, most of our systems are not redundant, so there will be a bunch of minor disruptions.

Stu, who is overseeing the build-out of what will largely be ‘his’ DC, showed me why Cat6A cabling is so much thicker (and thus harder to work with) than old-school Cat5 UTP (”Unshielded Twisted Pair”) — the internal copper wiring is twisted around itself many more times to reduce interference, and the whole thing is cradled by a plastic framework shaped like a plus sign. This framework is twisted as well, so as the Cat6A cables lay next to each other in cable trays, the individual conductor strands don’t align with neighboring Cat6A cables, again helping to avoid signal transference between what should be independent connections. The idea is that in 10 years, when everybody is demanding 10GE connections, we’ll be able to simply re-patch uplinks into 10GE switch ports as needed. Otherwise the rewiring would be painful for individual machines, and impossibly disruptive to do in bulk.

Unfortunately, the heavier-duty Cat6A is also heavier and bulkier, thus significantly harder to work with and slower to run. Each of the 24 new 42U racks is getting 48 runs, from 2 1U patch panels in each rack, back to 6 patch panels (96 connections) in each of the new network racks, where switches and other Cat5-based gear, such as terminal servers and KVM switches, will go. This is new 1,152 runs in addition to the slightly older stuff at the South end of the room, which is still our DR site during this construction.

My question is: How long will it be before we need more than 48 connections in a rack? Our non-blade Linux servers tend to have 3 Cat5 connections: Ethernet, serial console, and KVM; Windows systems don’t need serial consoles, so they get 2. A rack of 1U Linux servers maxes out at 40 1U servers and 120 Cat5 connections, which just won’t fly here. 8 2U Linux servers (24 connections) and 12 Windows servers (another 24 connections) fill a rack, meaning as time goes on and we are again someday tight for space, we might run out of network connections sooner. At that point we could put a KVM server in every third rack and reclaim a lot of cabling for Ethernet, but it violates our model of having everything run patched to the switch racks. We’ll see what the world looks like when we actually get there…

I discovered yesterday that they’re also simultaneously digging up the driveway between Founders Hall and Flexner — not sure why, but it looks like pipe-laying for plumbing.

Update According to Stu, this is actually conduit for electrical wiring, from the vault under our parking lot up through to an electrical switching station in Flexner.

Many more RU photographs are up at http://www.reppep.com/~pepper/album/ru/

Tuesday morning, Amy and I were walking to the subway together, and we saw this amusing sign:

I sent it to Heather & Sam, who run New York Shitty (she acid wit & poop snaps, he back-end hosting), and she liked it. Then Curbed picked it up, and it’s made the rounds.