Extra Pepperoni

To content | To menu | To search

Tag - people

Entries feed - Comments feed

Sunday, January 18 2009


For a while I kept off-site backups at Rockefeller, but I was never great about maintaining them.

CrashPlan attempts to make this simple and secure. Recently Sam asked me about CrashPlan, and we decided to host each other's backups. We each bought 1tb external drives, installed CrashPlan, and started kicking the tires. I noticed a few things.

The $60 CrashPlan+ license is per computer. Since CrashPlan automatically ties together all computers configured to share a personal account and the documentation talks about backing up between your computers, I thought the license was per account. $120 for a MBP and a Linux server is reasonable, but I was annoyed when I tried to install my license on the second computer and got a $60 error message.

The CrashPlanEngine grants control to anyone who can connect to it. This is bad, especially since my Linux server has a bunch of users on it.

Is it a bad idea to run the CrashPlanEngine as root? There are basic risks associated with it. These are the same as any service running as root. Because the engine is written in java, it is immune to buffer overflows, a common exploit for poorly written C code. You should be aware that any desktop client connecting has "permission" to select any file and back it up. Therefore, you should "require" authentication each time the UI is run. This is in the "settings/security" part of CrashPlan UI.

The installer failed to make /usr/local/crashplan/bin/CrashPlanEngine executable (fix: chmod +x /usr/local/crashplan/bin/CrashPlanEngine).

The installer only configured the CrashPlanEngine init script to run at runlevel 3. I might run at runlevel 5, and crashplan should still run, so I used chkconfig crashplan on.

Overall, I like CrashPlan so far. The UI is clean and includes everything I have needed. The design seems thoroughly reasonable. The encryption (stronger with CrashPlan+) gives me peace of mind. Seeding the initial backup over FireWire was fine -- we'll have to see real-world performance up our 768kbps DSL uplink.

Because we each have a significant amount of data, and moderate Internet connections at home, we are seeding locally -- performing the initial backups via FireWire & USB before exchanging drives. Afterwards, we'll update our own backups over the Internet -- CrashPlan supports this, and uses encryption to keep our data private, even from the partner who physically possesses the drive.

Our seeding is a bit complicated. Macs (my laptop & Sam's desktop) use HFS+. Linux (my server) uses ext3. FreeBSD (Sam's server) uses UFS2. This makes deciding what filesystems to put on the 1tb drives non-trivial, as they need to match the other person's CrashPlan server -- I will host Sam's drive on my Linux server, so want ext3; I don't know what format Sam wants, but hope it's not UFS2, as I no longer have a working FreeBSD server and Code42 doesn't support FreeBSD. CrashPlan has a doc on pre-seeding remote backups, but it's not terribly clear and assumes only 2 computers and a single filesystem format.

Monday, June 9 2008

This Must Be 2008 -- Blogs Are Everywhere!

When Amy mentioned to Joyce (of Scarce) that she now has a blog, Joyce was amazed and impressed at how cutting-edge Amy is. There's definitely a geographical factor here, because at my picnic earlier the same day, we figured out that of the 6 adults and Julia present (all Brooklynites), every single one of us has a blog.

Devjani's is firewalled. Julia's Journal runs on hand-crafted HTML rather than blogging software, but that's because it dates back to mid-2002; I will move it over at some point. Sharon has two. In addition to Extra Pep, I edit Securosis.

Thursday, June 5 2008

Childhood dreams fulfilled

Being the compulsive sort, it bugged me whenever I missed an episode of a TV show I watched (I used to watch a lot of TV; now not much). Similarly, it bothered me that I didn't have complete sets of the comics I read -- they were both hard to find and expensive, especially since I almost never started at the beginning.

Inspired by Ernie Cline, I've recently been watching Airwolf. It hasn't aged well, and was never great storytelling, but it's still enjoyable. And it's nice to see as a coherent whole over weeks, rather than scattered across years with commercial interruptions. I'm in the middle of season 2, and will skip season 4 (I don't think I ever saw it, fortunately); don't know about season 3. Perhaps I'll watch The Fall Guy next!

Nowadays, with the Internet, back issues of comic books are pretty easy to find. I've completed a few series that were missing issues, such as Badger crossovers, Dynamo Joe, and Tailgunner Jo. I'd love to collect various other series, but a full run of X-Men would be prohibitive -- both in terms of money and time to read them all!

I was pleased to discover Marvel made several of their more popular titles available to GIT, who released them on DVD. Unfortunately, the license was terminated in favor of Marvel's online service, but some DVDs are still available. James gave me Ghost Rider for my birthday, and despite some aggravations (they photographed the open comic books, so there's dead space around the corners, and didn't bother to split left & right pages, so it's too awkward to read in single-page portrait mode) which make the comic harder to read than it should be, I'm enjoying the old Ghost Rider issues. It's amazing what a loser Johnny Blaze originally was -- he's an idiot (sloppy writing), a coward, a regretful devil dealer, and not really faster or more skillful than gang members. As time has gone on, and Marvel has super-sized its characters, Ghost Rider and his cycle have gotten faster, stronger, less human, and ironically much more innocent.

Wednesday, June 4 2008

The Serious Shit

At Wheaton, I helped found the Progressive Alliance, a student political club. I don't remember most of the members (in fact I no longer recall the names of most of my classmates), but Kirsten Cappy was one of the heads -- one of two co-presidents, if I recall correctly -- and Steve Amster (a good friend to both of us) got me involved.

As the nerdiest Progressive, I ended up laying out The Serious Shit in PageMaker. Articles were of course always late, so I remember having to shorten articles I'd just stretched out to fill space, in order to fit post-deadline content onto the page (issues were one to two pages, letter or legal sized).

The Shit was posted on the bathroom stall doors, where we had a guaranteed audience with time to read. I don't recall much more about it, although if Jason Snell revives my old 210mb hard drive, I might get some old issues back -- unless they're on my 6 even older 44mb SyQuest cartridges.

The other thing I recall about TPA & TSS is that my mother convinced me that if I listed "Progressive Alliance" as an activity on my resume, people would decide I was a Communist and not hire me. I don't remember if I took her suggestion and called it "The Humanist Alliance", or simply left it out entirely. There was never any question of listing The Serious Shit on the resume -- I never interviewed for a job where that would have been a plus.

Fortunately, after my first job at Rockefeller University, I had more relevant things to put on my resume, so the Progressive Alliance dilemma quickly became a non-issue.

Monday, May 26 2008

Razor and CYA Idiocy

We got a Razor for Amy and me, so we can scoot with Julia. It's fun, but apparently for robust kids, as the handlebars are too low for a grownup, but it's rated up to 180 pounds. The handlebars bug me, though. They have a label which reads:

Caution: this moves when used. Exercise caution & common sense when riding.

Saturday, February 23 2008

Scarce at Union Hall

Joyce, singing

Joyce (Raskin) White is a friend of ours from the neighborhood -- Julia and her daughter Sydney are a couple days apart in age and were best friends when they lived in Brooklyn. A few years ago Joyce, Matt, and Sydney moved to Boston, and we were all sad. Before Brooklyn, Joyce was in a fairly successful rock band named Scarce, but they broke up after a brain injury took Chick Graining (lead singer) out of commission.

A couple years ago, Joyce started writing a book about her experiences growing up as a female rocker, called Aching to Be: A Girl's True Rock and Roll Story. Amy edited the book, and we've been waiting to see Scarce perform ever since.

Tonight they played at Union Hall, just down the street, and we finally got to watch Joyce rock out. It was most excellent, and I got a mess of pictures.

Monday, February 18 2008

System Admin Interview Questions

I was quite impressed by Joel's description of the hiring process, and we've been doing a lot of interviewing for System Admins lately. I put together a list of standard questions to ask during interviews, which has been quite helpful in judging a) how much technical knowledge people have, and b) (just as important) how good a match they are for the skills void we were trying to fill at the time. Here they are, for the next person who needs to perform a similar exercise.

  1. How many systems does your team manage (Linux, Solaris, Windows, etc.)?
  2. How large is your team?
  3. Which OS are you most comfortable/familiar with?
  4. Which Linux flavors are you most comfortable/familiar with?
  5. Which Red Hat versions are you familiar with?
  6. Are you familiar with kernel programming or configuration?
  7. Have you done any custom packaging or kickstarting?
  8. Have you used or managed Sun JumpStart?
  9. How much experience do you have with Sendmail?
  10. ... NetWorker? Version? Managing backups, or just configuring clients?
  11. ... LDAP? Brand & version? LDIF or just querying?
  12. ... firewalls (iptables, ipf, etc.)?
  13. ... network administration (Cisco, sniffing, etc.)?
  14. ... Apache httpd?
  15. ... Tomcat & Java?
  16. ... EMC (Clariion, PowerPath)?
  17. ... shell scripting, and with which shells?
  18. ... perl scripting?
  19. ... Veritas VM/FS? Versions?
  20. ... Veritas Cluster, or other HA? Versions?
  21. ... snapshots? In which products?
  22. ... load balancing
  23. ... Oracle (as SA, not DBA)?
  24. ... HPC?
  25. Please briefly explain the difference between RAID 1 and 5. What are layered RAID levels, and when are they appropriate?
  26. What sizable projects have you done recently?
  27. Why are you leaving your current employer / did you leave your last employer?
  28. Please give specific examples of some routine tasks you've performed recently.
  29. Have you done systems specification and design (servers, multi-server configurations)?
  30. Have you worked with customers directly, or primarily with/for other IT personnel?

It didn't make sense to publish a list of questions when I was involved in the interviewing process, but now that I'm leaving Rockefeller and no longer interviewing UNIX Admins for them, I can post my sample questions.

Friday, February 1 2008

Wiring Art

The Pretties

Inspired by When data center cabling becomes art from Andrew T Laurence & Chuck Goolsbee's pics of Digital Forest, I took some photos of Rockefeller's new data center. We've been planning out various scenarios for 5 years at this point, but we finally moved most of our systems in this month. Note that the network guys (mostly Eric) took care to run cables connecting to ports on the left half of each device in from the left, and come in from the right for ports on the right. This makes more work for them in preparation, since one cannot simply plug a cable into a free port, but makes things look prettier, and also reduces cable snarling. 3 KVMs & baby + LCD

More Connectivity, Please

Since we first started discussing data center plans, I've been saying we need more connectivity. The new DC has 48 patches per 42U rack, and some of the new racks are indeed running out of ports before they run out of vertical space. In our racks 2U is used for patch panels and 2 cables control APC managed power strips, so we have 40U and 46 patch ports for servers. Our Linux servers have Ethernet, serial console, & KVM; Suns have Ethernet & console; Windows have Ethernet & KVM. In the worst case, 40 1U Linux servers need 120 connections, but we only have 46 available. If the rack is full of 2U Suns & Windows servers, we're okay with 6 'extra', available for dual-connected servers or whatever. As we get more dense, we begin to run out of ports. Cat6 flowing down


Blades are no better -- their chassis tend to blow out the power budget because they're even more dense than 1Us (although they do get more servers per rack), and with all the redundancy they still require a lot of cabling. For a reasonable IBM BladeCenter, we need 4 x 2 for GE switches (FC cables don't go in these patch panels). Then 2 x 2 for (Ethernet & KVM) for management modules per chassis = 12 ports for 7U. For our new HP c7000 chassis with basic networking, we have 16 GE ports, 2 GE console ports, 2 OA Ethernet ports, and 2 2 OA serial ports (again, ignoring the fiber-optic GE ports): 22 ports in 10U. I'm sure somewhere HP has demo chassis, filled them with fully-connected GE switch modules: (9 x 8 + 4 = 74 patches) & (4 x 8 = 32 fiber-optic ports) = 106 cables total (not counting power connections -- 6 in our case). In 10U -- 1/4 of a rack -- insane! c7000: 30 ports

Update 2008/2/5: Eric pointed out I was wrong about the ports -- the Cisco switches have 8 uplink ports, 4 of which are either fiber-optic or copper (you can see they're 17-20 in the photo); the other 4 copper ports seem intended for cross-linking to the other switch. So the max copper patch count remains, but the the fiber connections would be instead, rather than in addition, and we may fully connect our 2 switches with only 8 GE uplinks rather than 16 going out of the chassis.

Friday, January 4 2008

Twitter Is ...

Glenn hung up his Tweets a few days ago, which makes this an apropos time to ponder Twitter and whether it's worthwhile.

  • Micro-blogging.
  • Super chat status line messaging.
  • Perhaps the easiest way to flex your vanity, Web 2.0 style.
  • A chat room without walls.
  • A remarkably uninformative way to exercise your vanity, since one has no idea how many people might read a tweet.
  • A particularly 21st-century game of one-upsmanship -- # followers vs. # following.
  • A pleasant diversion / horrendous distraction.
  • An excellent way to broadcast information, including emergency notifications, although its lack of pervasiveness limits what it's good for.
  • Access to lazyweb.

Friday, December 21 2007

Music at Jalopy: Gavin Smith, Anna Copa Cabanna, & Royal Pine

I went out tonight to see Gavin perform (accordion, piano, & backing vocals) with Anna Copa Cabanna at Jalopy, a performance space and instrument repair shop so old-timey and rustic they had a wooden bolt to lock the bathroom door and a wooden box sink.

Getting there was surprisingly difficult. I called a local car service, and they told me it would be 10 minutes. In 6, the car was outside; when I went out, she complained that I'd kept a woman waiting, and spent much of the drive complaining about the car we were in. Then she drove 7 blocks south to 9th Street, and asked me how to get there. Since the driver didn't know the way, I used Google Maps on the iPhone to give her directions. As we got to the Fort Hamilton Parkway, she began to tell me that Columbia Street (which Jalopy is on) didn't exist on the other (north) side of the Parkway; I insisted at least half a dozen times, and guided her around the entrance -- insisting to me all along that we couldn't get to the north side of the Parkway, until we saw the place, right where Google claimed it was. To get home I called Eastern, and they were 30% cheaper without arguing or needing directions.

I got some very dark pictures (no flash).


It was a very good show, although as a friend of Gavin's I didn't really appreciate the harassment of "Smitty" that's part of their shtick. On the other hand, I wouldn't have appreciated him treating her that way either...

Lots of strange songs about New York and America, by a fascinating and deliberately somewhat crazy outsider (Australian import). I was particularly impressed that they managed to do "Beauty Bar" as a real punk song -- on tambourine & accordion.

Anna & the MG5

I also enjoyed Royal Pine, but was a bit freaked out when they played a song I recognized -- "Pearl Polly Adler". This is inexplicable because I listen to perhaps a couple dozen songs I don't know each year (largely from James), so recognizing a 'new' song that's not in my iTunes library was a (pleasant) source of shock and confusion. Apparently I found it while surfing YouTube recently -- something I also very rarely do, as opposed to effectively never before the iPhone. The Pearl Polly Adler video looks familiar, but I can't be sure it's not all delusional deja vu.

Anyway, I enjoyed that song while racking my brain for where I had heard it before, and "Stone Cold Mamacita" even more, although I enjoyed the rougher and tougher live version more than the recording on Huasteca.

Tuesday, December 4 2007

Holiday Albums

I take a lot of pictures of Julia, and every year we make holiday photo albums (normally from iPhoto); last year we got 6.

I just went through December 2006's photos, picking 5. Now I have 2,400 that made the initial cut from January through November 2007 to review. There are also 47 Julia took this year to check out.

It's a big job! The books tend to be a bit longer than the base 20 pages, but we like them.

Thursday, October 4 2007

New uses for passwords

I was walking down the street this morning, burning a piece of paper with some old passwords on it, and holding the box of matches I had used to light it. A woman saw me, and said "Hi. Gimme a match?" I got out a match and prepared to light it for her. Before I could strike flame, the woman leaned over to my burning password paper and lit a cigarette from it, then said "Thank you."

There I was, standing on the street, thinking "Smoking's bad, mm-kay," and wondering why she asked for a match when she wanted a light (yes, I know, I cannot turn off being an editor), and thinking this was probably actually not the first time someone's lit a cigarette from a burning password, but it's still unusual.

Wednesday, October 3 2007

Old School: Ancient UNIX

Rockefeller University, where I work, was one of the original UNIX sites. In 1975, Mel Ferentz held what was apparently the second UNIX users group meeting (it is not clear if he was one of the organizers of the first meeting, in 1974). Mel went on to build USENIX out of those meetings. He moved on to Rockefeller University soon after those first meetings; just before I started at RU Computing Services, Mel stepped down as Director of RUCS, and moved on to develop Internet2 at NYSERNET.

Last week, Mark Kowitz left RU IT (RUCS after a name change), where he had worked for 23 years. Mark met his wife, Robin, in RUCS over 20 years ago. I met Amy there too, when I started in 1992 (I left in 1995, and Amy left in 1996; I came back; she has not). While cleaning out his papers, Mark found some old documentation on booting UNIX on the PDP-11/70, VAX 11/750, and VAX 11/780, and passed it along to me. Mark doesn't remember whether he or Mel wrote the documentation, but it is visibly classic UNIX documentation (distinctive fonts and layout).

Ancient UNIX boot instructions

You can see some more about booting PDP-11 UNIX (in emulation) at Ancient UNIX, 8bitsunplugged.org Digital Archeology, and Amit Singh's GBA UNIX.

To give you some idea of how much water there has since been under this particular bridge, UNIX was first developed on a DEC PDP-7 in 1969. Digital Equipment Corporation was basically bought by Compaq, which itself was later acquired by HP. This version of UNIX contains Western Electric license statement; UNIX was createdat Bell Labs, which was jointly owned by Western Electric and AT&T. Bell Labs was later absorbed into AT&T, spun out as part of Lucent, and merged with Alcatel to become part of Alcatel-Lucent.

AT&T split off UNIX into UNIX System Labs, which was later bought by Novell. Novell sold much of the UNIX business to Santa Cruz Operation, which sold its UNIX rights and the "SCO" name to Caldera. SCO changed its name to Tarantella and Caldera transformed itself from a Linux company into a UNIX company named "SCO Group". Alas, Caldera didn't make money either way, and eventually sued the world -- IBM, Novell, various of its own customers, etc.

Along the way, several BSDs were created to provide an alternative to AT&T's UNIX, later providing a family of excellent UNIX-based operating systems (including the core of Mac OS X). In contrast, Linux was launched in 1991 by Linux Torvalds, born in 1969, the same year as UNIX.

Those little pages are quite a time capsule!

Another paper, by Dennis Richie: http://cm.bell-labs.com/cm/cs/who/dmr/cacm.html.

Sunday, September 16 2007

Harry Potter and the Deathly Hallows

Just finished Harry Potter (thanks, James!). We're looking forward to reading them to Julia in a few years.

Friday, August 3 2007

Lots of Construction on Campus

The Super-Tent/IT Pavilion/Big Top/Big House fronts on the main RU parking lot, at the other end of which is the 66th St Gate. Except that after we moved in, they walled in the lot and started digging:

Parking Lot and Super-Tent

They still haven't started on Smith Hall, though, which makes me wonder why we couldn't still be in a proper building now. In the meantime, the main campus entrance and driveway are closed, along with the parking lot, under which a new electrical vault will be built. Getting around campus is much more complicated now than 6 months ago. This is especially true for IT, moving equipment around the tent, as the pathways and steps around the periphery don't quite work for carts.

Our new main data center is nearing completion. It was previously our backup/disaster recovery site, so needed a lot of build-out to fit the rest of our servers. The swap from the older/smaller UPS system to the newer/larger one will be tricky, as several live servers will be switched over while running. Later we get to swap systems end-for-end across campus, so the primaries are in the primary DC, once their current location becomes the DR site. Needless to say, most of our systems are not redundant, so there will be a bunch of minor disruptions.

Stu Cohnen

Stu, who is overseeing the build-out of what will largely be 'his' DC, showed me why Cat6A cabling is so much thicker (and thus harder to work with) than old-school Cat5 UTP ("Unshielded Twisted Pair") -- the internal copper wiring is twisted around itself many more times to reduce interference, and the whole thing is cradled by a plastic framework shaped like a plus sign. This framework is twisted as well, so as the Cat6A cables lay next to each other in cable trays, the individual conductor strands don't align with neighboring Cat6A cables, again helping to avoid signal transference between what should be independent connections. The idea is that in 10 years, when everybody is demanding 10GE connections, we'll be able to simply re-patch uplinks into 10GE switch ports as needed. Otherwise the rewiring would be painful for individual machines, and impossibly disruptive to do in bulk.

Unfortunately, the heavier-duty Cat6A is also heavier and bulkier, thus significantly harder to work with and slower to run. Each of the 24 new 42U racks is getting 48 runs, from 2 1U patch panels in each rack, back to 6 patch panels (96 connections) in each of the new network racks, where switches and other Cat5-based gear, such as terminal servers and KVM switches, will go. This is new 1,152 runs in addition to the slightly older stuff at the South end of the room, which is still our DR site during this construction.

My question is: How long will it be before we need more than 48 connections in a rack? Our non-blade Linux servers tend to have 3 Cat5 connections: Ethernet, serial console, and KVM; Windows systems don't need serial consoles, so they get 2. A rack of 1U Linux servers maxes out at 40 1U servers and 120 Cat5 connections, which just won't fly here. 8 2U Linux servers (24 connections) and 12 Windows servers (another 24 connections) fill a rack, meaning as time goes on and we are again someday tight for space, we might run out of network connections sooner. At that point we could put a KVM server in every third rack and reclaim a lot of cabling for Ethernet, but it violates our model of having everything run patched to the switch racks. We'll see what the world looks like when we actually get there...

I discovered yesterday that they're also simultaneously digging up the driveway between Founders Hall and Flexner -- not sure why, but it looks like pipe-laying for plumbing.

Trench between Founders and Flexner Update According to Stu, this is actually conduit for electrical wiring, from the vault under our parking lot up through to an electrical switching station in Flexner.

Many more RU photographs are up at http://www.reppep.com/~pepper/album/ru/

Wednesday, June 20 2007

Daring Fireball Visits NYC

John Gruber (Daring Fireball) gave a presentation at the SoHo Apple Store tonight. I might've been annoyed it was a repeat of his C4[0] presentation, except I wasn't at C4 so I hadn't heard it. The rest of the audience seemed suitably impressed -- Apple brought extra chairs, and there were still a bunch of people sitting on the floor.

Afterwards, I tagged along to a yummy Vietnamese restaurant. We left when all the unoccupied chairs had been placed on the tables around us, only to discover a giant (empty) drum of MSG outside the front door. This sparked a brief but lively discussion of whether MSG is in fact as bad for you (us) as people once claimed, with no real resolution.

I liked Gruber's description of Jonas fuzzing, "I need a hole."

Thursday, January 18 2007

Securosis Will No Longer Cover Technology

Rich has been told to stop blogging about techonology. This is a shame, as he had worthwhile things to say.

Having very little information on what happened, I have to assume it's a blanket policy intended to protect Gartner's intellectual property, by reducing competition from non-Gartner IP (such as public blogs). I wonder how bad the backlash will be. This is aside from the fact that Rich was a) careful not to post Gartner content and b) not shy about mentioning what you could get if you were a Gartner client.

It's a pity.


Thursday, November 23 2006

Optional Sidebars with Apache SSI

I'm writing online documentation for a site that has substantial left-side nav in a sidebar: http://www.xowave.com/. We want to be able to hide the sidebar, perhaps to build a smaller tarball of the docs, or to make more room for large images on smaller screens, or to save space on crowded pages.

Bjorn (the developer) made the sidebar conditional a few days ago, and I just enhanced it this morning to provide a user-accessible knob (in footer.incl , so on available on every normal page) to flip it on and off. Additionally, with "wget --user-agent=printme", we can whack the whole site without nav. I don't actually want to do this, but it's a nice feature.

I actually tested with something like "curl --user-agent printme URL | grep -i agent", using the URL of a special test.shtml page that basically just contained <!--#printenv -->. It was very handy for figuring out what the server thought of my requests.

footer.incl contains this snippet:

<!--#if expr="$QUERY_STRING = printme" -->
    <a href="<!--#echo var="SCRIPT_URI" -->">Restore navigation sidebar</a>
<!--#else -->
    <a href="<!--#echo var="SCRIPT_URI" -->?printme">Hide navigation sidebar</a>
<!--#endif -->

And head.incl wraps the sidebar code in:

<!--#if expr="$sidebar = hidden || $QUERY_STRING = printme || $HTTP_USER_AGENT = printme"-->