Extra Pepperoni

To content | To menu | To search

Friday, November 20 2015

Austin BBQ

Achievement unlocked I flew to Austin, TX for the SC15 conference. I had two personal goals: to eat BBQ for lunch or dinner every day, and to stay within Austin. Aside from visits to small towns around Austin (Wimberly for crafts, Johnson City for LBJ's home, and Driftwood for the Salt Lick), I accomplished my goals. I asked friends for recommendations and got a few; over 7 days I had BBQ 9 times, from 8 places. The only failure was Franklin.

Austin Convention Center

Salt Lick

  • Tue: Brazilian all-you-can-eat meat (steak) dinner, then Rudy's takeout (token brisket) at another party.

  • Wed: la barbecue (lunch, uncrowded), with dinner at the Alamo Drafthouse (Spectre @ the Ritz)

la barbecue ritz

micklethwait 1 blacks stadium

  • Fri: We arrived at Franklin BBQ at 10:40, and the line looked reasonable, but the hostess told us that we'd get food at 2pm. This was absurd and too late for our 2:30 flight, so we walked to Micklethwait for another excellent lunch (first order when they opened at 11am).

franklin micklethwait 2


Friday, April 4 2014

The Doubleclicks at The Way Station, with Sarah Donner, April 4, 2014


Sarah Donner opened:

  1. Rogue
  2. Heartbreaker
  3. Bitches You Can Steal My Shit
  4. The Motherfucking Pterodactyl (see also the Oatmeal's great animated video)
  5. The Rebuttal of Schroedinger's Cat (no video available)
  6. The Kickstarter Song
  7. With Pride
  8. Signal
  9. Going under (in a Way)
  10. All My Guns

Then The Doubleclicks played:

  1. Will They or Won't They
  2. Worst Superpower Ever
  3. Cats and Netflix
  4. Lasers and Feelings
  5. A Lullaby for Mr. Bear (adult version)
  6. Wonder (Wonder Woman Song)
  7. Ennui (on We Go)
  8. Nothing to Prove
  9. Something Else
  10. This Fantasy World
  11. Velociraptor (?)
  12. I Love You Like a Burrito (with special guest)

Saturday, March 29 2014

4th St. NiteOwls, Freddy's, March 29th, 2014

Gavin played with the NiteOwls at Freddy's.


  1. Jack I'm Mellow
  2. Breaking the Ice
  3. Busy Bee
  4. A Sin to Tell a Lie
  5. When Day is Done
  6. Louisiana Fairy Tale
  7. Crazy about My Baby
  8. A Porter's Love Song to a Chambermaid
  9. Nagasaki
  10. Bloodshot Eyes
  11. Diga Diga Do
  12. Honeysuckle Rose
  13. Jerry the Junker
  14. Happy Birthday

Wednesday, June 26 2013

Solaris patching is broken because Oracle is dumb and irresponsible

I am setting up a Solaris 10 system, starting from S10U10 (Solaris 10 Update 10) as a starting point to match another server. Solaris includes a registration wizard that comes up automatically after installation, but it doesn't work. Oracle updated their whole online patching system when they took over from Sun and broke the old built-in patching tools. Unfortunately the procedure to update an old system is completely byzantine.

You have this problem if the graphical Solaris Registration Wizard says "Error in SCN/Cacao Update License" when you register, or the smpatch command errors out like this:

-bash-3.2# smpatch analyze
Error: Unable to download document : "xml/motd.xml"
Cannot connect to retrieve motd.xml: Authorization Required
Failure: Cannot connect to retrieve current3.zip: Authorization Required
-bash-3.2# cat /etc/release
Oracle Solaris 10 8/11 s10x_u10wos_17b X86
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Assembled 23 August 2011

If you kept your Solaris system patched during the transition period you were presumably fine, as hopefully they released client updates before the took down the old backend, but old systems like my new install get stuck.

I opened a case with Oracle Support who searched their internal database and gave me an irrelevant answer. The real fix is to run sconadm manually -- which entails manually creating a (simple) RegistrationProfile.properties file per the instructions in the sconadm(1M) manual page, embedding your Oracle Support username and password in the file because Oracle cannot do security, registering, and then immediately deleting the file.

Then smpatch installed a bunch of patches and choked on 147993-05 SunOS 5.10_x86: Pidgin libraries patch. The patch instructions say to install SUNWgnome-im-client-root from the installation DVD. Our X4500s don't actually have DVD drives -- why can't I just download this package from https://support.oracle.com/? It turns out SUNWgnome-im-client-root is not on sol-10-u10-ga2-x86-dvd.iso, but it is available on sol-10-u11-ga-x86-dvd.iso.

-bash-3.2# smpatch update -i 147993-05
Installing patches from /var/sadm/spool...
Failed to install patch 147993-05.

Utility used to install the update failed  with exit code 15.
Validating patches...Loading patches installed on the system...Done!Loading patches requested to install.Done!The following requested patches have packages not installed on the systemPackage SUNWgnome-im-client-root from directory SUNWgnome-im-client-root in patch 147993-05 is not installed on the system. Changes for package SUNWgnome-im-client-root will not be applied to the system.Checking patches that you specified for installation.Done!Approved patches will be installed in this order:147993-05 Checking installed patches...Executing prepatch script...No SUNWgnome-im-client-root package can be found. The SUNWgnome-im-client-rootpackage must be installed before applying this patch.Please see the patch README NOTE 1 for information on installing SUNWgnome-im-client-root.The prepatch script exited with return code 1.Patchadd is terminating.

Jun 26 09:54:09 dhcp-172-21-230-215 root: [ID 702911 user.alert]  => com.sun.patchpro.util.PatchBundleInstaller@1342a67 <=Failed to install patch 147993-05.
Failed to install patch 147993-05.
ALERT: Failed to install patch 147993-05.
/var/sadm/spool/patchpro_dnld_2013.06.26@09:54:03:EDT.txt has been moved to /var/sadm/spool/patchproSequester/patchpro_dnld_2013.06.26@09:54:03:EDT.txt

Then I just had to manually install one more patch which smpatch refused to, and I was current. Good Times(TM)!

Thursday, March 21 2013

iptables: connlimit

Today we were effectively subjected to a DDoS attack by a badly behaved client raiding one of our web servers. We decided to rate-limit HTTP connections, which turned out to be pleasantly simple.

I replaced our old iptables rule to allow HTTP connections:

-A INPUT -j ACCEPT -p tcp --dport    22

with 2 new rules:

-A INPUT -j ACCEPT -p tcp --dport    80 -s xxx.yyy.0.0/16 --syn -m connlimit ! --connlimit-above 20
-A INPUT -j ACCEPT -p tcp --dport    80                   --syn -m connlimit ! --connlimit-above 5 --connlimit-mask 24

The first rule allows any on-campus user to make 20 simultaneous connections. The second rule prevents external clients from making more than 5 at a time. They can make as many connections as they want in series, but if a web crawler attempts to open 6 connections without closing any, the 6th will time out. Slick!

Thanks to nixcraft for the details!

Friday, February 15 2013

Jill Sobule with Sex Mob at Union Hall, 2/15/2013

Colin McGrath opened. The oddest moment of the night was when he told us he had never heard Jill before, but sound check was great.

Jill played with Sex Mob, who she apparently first heard at the old Knitting Factory and is now touring with. They were excellent.


  1. Palm Springs
  2. Barren Egg
  3. When They Say We Want Our America back, What the F#¥k Do They Mean?
  4. Joey
  5. I Kissed a Girl
  6. Kickstarter rap
  7. Filthy Little Devils
  8. Raleigh Blue Chopper
  9. Angel/Asshole
  10. Smoke Dreams
  11. Cinnamon Park
  12. Rocky Mountain Way
  13. Sunrise Sunset
  14. Resistance Song
  15. Modern Drugs
  16. Jetpack
  17. Nothing to Prove
  18. Steve's joke
  19. Attic
  20. Jill's joke (her first ever onstage?)
  21. Lucy at the Gym
  22. When My Ship Comes in
  23. Lucky in Love

Tuesday, January 22 2013

Dell DRAC & Macs

We have been vexed by Dell's lack of Mac compatibility for some time, but things have gotten better lately. Our problems are specifically with DRAC's virtual console feature.

The virtual console on our R910 (iDRAC 6) was incompatible with the Mac keyboard layout. We could bring up a virtual console but the characters on-screen did not match keys pressed. Our workarounds were a) to connect to a Linux system via VNC and run Firefox there, or b) to connect from a Linux or Windows VM running on the Mac. Imagine my surprise when I tried last week and the keyboard worked properly! We haven't updated the iDRAC firmware on that system lately, so presumably this was fixed in Java, but it's a major improvement.

Second, when I tried to bring up the the virtual console on an R820 I got an uninformative timeout. Some helpful Dell folks pointed me to v1.30.30 of the iDRAC 7 firmware, which adds Mac Safari compatibility.

To get v1.30.30, search for your service tag on http://support.dell.com/, select ESM (Embedded Systems Management), and choose the 'Application' .exe download. Only choose one item to download, or Dell requires you to use Akamai's download manager, which doesn't work for me. The .exe file appears to be a self-extracting .zip archive, so just unzip it and feed firmimg.d7 to the iDRAC 7 Firmware Update page.

Friday, January 11 2013

Blowhole Theater Winterlude 2013 at Barbes

The Blowhole Theater Winterlude 2013 was at Barbes January 10th.


Ditty Committee played:

  1. Unicorn Farts
  2. Grand Larceny
  3. Your Drunk Wife
  4. The Ditty Committee Almost Knows the Alphabet
  5. Meat & Money, Inc.
  6. Ice Cream Truck
  7. Song of the Week
  8. Landfill Harmonic

Ed Pastorini played a few songs with vocal assistance from Don Ralph. I didn't catch any titles, unfortunately.

Virpi arrived just in time to participate in a Blowhole ritual.

Don Ralph, Gavin Smith, & Susan Hwang performed 3 pieces:

  1. The Whole Thing down
  2. Foul Pies
  3. Mobile Wash Unit

Thursday, September 20 2012

4th Street NiteOwls at Barbes, 2012/09/20

Gavin played clarinet with the NiteOwls tonight.



  1. Diga Diga Do
  2. Jack I'm Mellow
  3. Honeysuckle Rose
  4. Breakin' the Ice
  5. Crazy 'bout my Baby
  6. Making Whoopie
  7. Have a Little Dream on Me
  8. Bloodshot Eyes
  9. Louisiana Fairy Tale
  10. Jerry the Junker
  11. Ninety One in the Shade
  12. Staying Alive
  13. Nagasaki
  14. Cross Patch
  15. When Day is Done
  16. It's a Sin to Tell a Lie
  17. I Wish I Were Twins

Previous shows:

Saturday, May 19 2012

James & Sharon Gerber

Our good friends James & Sharon got married Saturday in Rhinecliff, NY. It was a beautiful day with lovely people. The official photos by Alex Fedorov aren't up yet, but my photos are up (thank you Rich, Julia, & Amy for assistance).

Ask me or the Gerbers for the password. You can also upload photos to this gallery (with a different URL & password).


My toast:

I have known James almost 30 years, since he gave me a tour of his high school and convinced me to go there. In retrospect I see that tourguide role as early days for an outgoing and friendly actor. In high school James and I became closer as members of a fairly tight-knit group (cough gamers), which stayed together through college and as we returned to NYC afterwards. But James is the only one I'm still in regular contact with -- partially because he is still in town, but really because he's a great friend.

A few weeks ago I told Julia (hi Julia!) that I was having lunch with James. She told me that everybody could use more time with James, and she wasn't even mad at me for seeing James when she had to be in school. Thanks, kiddo!

I still remember the day James told me that he had made some comment about "A good day to die" or perhaps it was "A good day to move!" to a girl, and she had texted back "k'plah!" James and I agreed: this girl was a keeper. That was several years ago, but Sharon has proven to be much more than just a pretty face with a passing knowledge of Klingon.

Whenever a mutual friend asks how James is doing, I say he seems to have found his ideal match, and it's great to see how good they are for each other. We wish you guys all happiness!

As parents, Amy and I take turns going to grown-up parties like birthdays. Whenever I go to a James & Sharon thing, Amy asks me how everything went. I tell her it was nice to see today's stars, of course [nod], and that their friends are consistently interesting and pleasant to hang around with. I'm sure there's a quote about knowing the quality of someone's character by their friends, but I can't recall it. But thank you guys for all being a good bunch, too!

James, Sharon, we love you guys. We are so happy to see you together.

Friday, May 4 2012

MC Frontalot

As a Hannukah/Christmas present, Amy took me to see MC Frontalot at Knitting Factory Brooklyn. We were impressed by Schaffer the Darklord, fled the unintelligible noise of Math the Band, and didn't stay for the Wheatus, the headliners. In preparation, I played a bunch of Frontalot for Amy, which was useful -- it was much easier to enjoy music where we knew the words, as they were often difficult to hear.

The show was a success -- Amy is now a fan.

Friday, April 27 2012

Isilon Notes, 2012 Edition


  • Isilon provides templates for Nagios, which you should use. Unfortunately Nagios cannot distinguish serious problems (failed disk) from trivia (quota violations & bogus warnings).


  • Isilon's current units are either 2U (12-bay 200 series) or 4U (36-bay 400 series).
  • The new NL400-108 nodes are similar enough to the older 108NL nodes that they pool together. The 108NLs are dual-socket 16gb nodes based on the 72000x chassis, which is an upgrade from the 36000x chassis. This makes them much faster than the older single-core 36NLs & 72NLs.
  • As of OneFS v6.0(?), Isilon nodes no longer use the VGA keyboard & mouse console. Instead they use the serial port exclusively as console, although the VGA port does display some booting messages. In 2011, a USB connection to a KVM made a node reboot until we disconnected USB.
  • Every node is assigned a device ID when it is joined to the cluster. All alerts are tagged with the device ID of the node reporting the event. Device IDs are never reused, so if a chassis fails and is swapped out, the replacement will get a new device ID, but the old node's hostname. If this happens to you, you may want to use isi config (with advice from Isilon Support) to change the hostname to match the device ID. With a large or dynamic cluster it might just be better to ignore device IDs and let the node names run in a contiguous sequence.


  • Isilon's job engine is problematic. Only one job runs at a time, and jobs are not efficiently parallelized.
  • MultiScan combines Collect and AutoBalance jobs.
  • During the Mark phase of Collect (or MultiScan), with snapshots enabled, delete is slow and can cause NFS timeouts.
  • It is fine for non-disruptive jobs to run in the background for long periods, and it is understandable for high-priority jobs to briefly impact the cluster, but there are too many jobs (SmartPools, AutoBalance, Collect, MultiScan) which have a substantial impact on performance for long periods.
  • There are enough long-running jobs that it's easy to get into a cycle where as soon as one finishes another resumes, meaning a job is always running and the cluster never actually catches up. It took months for us to get this all sorted out so the jobs run safely in the background and don't interfere badly.
  • When a drive does not respond quickly, Isilon logs a 'stall' in /var/log/messages. Stalls trigger "group changes", which can trigger jobs. Group changes also disrupt jobs including MultiScan, AutoBalance, & MediaScan from completing. The workaround is to tune /etc/mcp/override/sysctl.conf per Isilon Support.
  • The default job priorities were dysfunctional for us. We had to alter priorites for AutoBalance, SnapshotDelete, SmartPools, and QuotaScan, and frequency for at least SmartPools. This improved somewhat in v6.5.
  • To tweak job priority, do not redefine an existing priority. This caused problems as the change cascaded to other jobs. Define a new priority instead.

Batch Jobs

  • /etc/mcp/templates/crontab is a cluster-wide crontab; field #6 is username.

Support & Diagnostics

  • By default, Isilon's main diagnostic command, isi_gather_info, builds a tarball of configuration and logs and uploads it to EMC. This took over 15 minutes on our clusters. To make this quicker, change "Gather mode" to Incremental under Help:Diagnostics:Settings.
  • Isilon does not actually maintain an HTTP upload server, so uncheck HTTP upload to avoid a wasted timeout.
  • When a node crashes it logs a core in /var/crash, which can fill up. Upload the log with 'isi_gather_info -s "isi_hw_status -i" -f /var/crash' on the affected node before deleting it.

Network & DNS

  • Isilon is "not compatible" with firewalls, so client firewalls must be configured to allow all TCP & UDP ports from Isilon nodes & pools back to NFS clients (and currently SNMP consoles).
  • Specifically, there is a bug where SNMP responses come from the node's primary IP. iptables on our Nagios console dropped responses which came from a different IP than Nagios queried.
  • To use SmartConnect you must delegate the Isilon domain names to the SmartConnect resolver on the cluster. We were unable to use DNS forwarding in BIND with this delegation active.


  • By default Isilon exports a shared large /ifs filesystem from all nodes. They suggest mounting with /etc/fstab options rw,nfsvers=3,rsize=131072,wsize=524288.


  • Migrating an IP to another node disconnects CIFS clients of that IP.
  • CIFS clients should use their own static SmartConnect pools rather than connecting to dynamic SmartConnect pools (for NFS clients).

Load Balancing

  • Rather than real-time load balancing, Isilon handles load-balancing through its built-in DNS server (SmartConnect: Basic or Advanced). Because this happens at connection time, the cluster cannot manage load between clients which are already connected, except via "isi networks --sc-rebalance-all", which shuffles server-side IPs in to even out load. Unfortunately OneFS (as of v6.5) does not track utilization statistics for network connections, so it cannot intelligently determine how much traffic each IP represents. This means only Round Robin and Connection Count are suitable for "IP failover policy" (rebalancing) -- "Network Throughput" & "CPU Usage" don't work.
  • High availability is handled by reassigning IPs to different nodes in case of failure. For NFS this is seamless, but for CIFS this causes client disconnection. As a result CIFS clients must connect to static pools, and "isi networks --sc-rebalance-all" should never be run on clusters with CIFS clients (there is apparently a corresponding command to rebalance a single pool, suitable for manual use on each dynamic pool).


  • Some of the advantage of the single filesystem is lost because it is impossible to move files from one quota under another. This forces us to copy (rsync) and then delete as if each quota were its own mount point.
  • For user quota reporting, each user should have an account (perhaps via LDAP or AD) on the cluster.
  • For user quota notifications, each user must have an email mapping (we created aliases to route machine account quota notifications to the right users).


  • The user Enable checkbox disables all login access (but preserves UID mappings for quota reports). Unchecking it blocks both ssh and CIFS/SMB access and clears the user password.
  • You cannot create a user with a home directory that exists (even with --force). Workaround: move the directory aside before creating the user, or create with a bogus homedirectory (which can only be used once) and use "isi auth local user modify" to fix after creation.
  • Don't use more than 8 SyncIQ policies (I don't know if this bug has been fixed).
  • Gateways and priorities are not clear, but if there are 2 gateways with the same priority the cluster can get confused and misbehave. The primary gateway should have the lowest priority number (1).
  • We heard one report that advisory quotas on a SyncIQ target cluster caused SyncIQ errors.
  • If you configure two gateways with the same priority, the cluster can get confused and misbehave.
  • In at least one case, advisory quotas on a SyncIQ target disrupted SyncIQ.
  • The Virtual Hot Spare feature appears to reserve twice as many drives as are specified in the UI, and they do not work as described.


  • Support is very slow. SLAs apparently only apply to parts delivery -- our 4-hour service does not prevent Isilon from saying they will answer questions in a few days.
  • Support is constantly backlogged. Callback times are rarely made and cases are often not followed up unless we call in to prod Support.
  • My process for opening a case looks like this:
    1. Run uname -a; isi_hw_status -i; isi_gather_info.
    2. Paste output from first 2 commands and gather filename into email message.
    3. Describe problem and send email to support@.
    4. A while later we get a confirmation email with a case number.
    5. A day or two later I get tired of waiting and phone Isilon support.
    6. I punch in my case number from the acknowledgement.
    7. I get a phone rep and repeat the case number.
    8. The phone rep transfers me to a level 1 support rep, who as a rule cannot answer my question.
    9. The L1 rep tries to reach an L2 rep to address my question. They are often unable to reach anyone(!!!), and promise a callback as soon as they find an L2 rep.
    10. As a rule, I do not receive a callback.
    11. Eventually I give up on waiting and call in again.
    12. I describe my problem a third time.
    13. The L1 tech goes off to find an answer.
    14. I may have to call back in and prod L1 multiple times (there is no way for me to reach L2 directly).
    15. Eventually I get an answer. This process often takes over a week.
  • Support provides misinformation too often. Most often this is simple ignorance or confusion, but it appears to be EMC policy to deny that any problem affects multiple sites.


For manual pages, use an underscore (e.g., man isi_statistics). The command line is much more complete than the web interface but not completely documented. Isilon uses zsh with customized tab completion. When opening a new case include output from "uname -a" & "isi_hw_status -i", and run isi_gather_info.

  • isi_for_array -s: Execute a command on all nodes in in order.
  • isi_hw_status -i: Node model & serial number -- include this with every new case.
  • isi status: Node & job status. -n# for particular node, -q to skip job status, -d for SmartPool utilization; we use isi status -qd more often.
  • isi statistics pstat --top & isi statistics protocol --protocol=nfs --nodes=all --top --long --orderby=Ops
  • isi networks
  • isi alerts list -A -w: Review all alerts.
  • isi alerts cancel all: Clear existing alerts, including the throttled critical errors message. Better than the '''Quiet''' command, which can suppress future errors as well.
  • isi networks --sc-rebalance-all: Redistribute SmartConnect IPs to rebalance load. Not suitable for clusters with CIFS shares.
  • du -A: Size, excluding protection overhead, from an Isilon node.
  • du --apparent-size: Size, excluding protection overhead, from a Linux client.
  • isi devices: List disks with serial numbers.
  • isi snapshot list --schedule
  • isi snapshot usage | grep -v '0.0'
  • isi quota list --show-with-no-overhead | isi quota list --show-with-overhead | isi quota list --recurse-path=/ifs/nl --directory
  • isi quota modify --directory --path=/ifs/nl --reset-notify-state
  • isi job pause MultiScan / isi job resume MultiScan
  • isi job config --path jobs.types.filescan.enabled=False: Disable MultiScan.
  • isi_change_list (unsupported): List changes between snapshots.
  • sysctl -n hw.physmem: Check RAM.
  • isi device -a smartfail -d 1:bay6 / isi devices -a stopfail -d 1:bay6 (stopfail is not normally appropriate)
  • isi devices -a add -d 12:10: Use new disk in node 12, bay 10.
  • date; i=0; while [ $i -lt 36 ]; do isi statistics query --nodes=1-4 --stats=node.disk.xfers.rate.$i; i=$[$i+1]; done # Report disk IOPS(?) for all disks in nodes 1-4 -- 85-120 is apparently normal for SATA drives.
  • isi networks modify pool --name *$NETWORK*:*$POOL* --sc-suspend-node *$NODE*: Prevent $POOL from offering $NODE for new connections, without interfering with active connections. --sc-resume-node to undo.
  • isi_lcd_d restart: Reset LEDs.
  • isi smb config global modify --access-based-share-enum=true: Restrict SMB shares to authorized users (global version); isi smb config global list | grep access-based: verify (KB #2837)
  • ifa isi devices | grep -v HEALTHY: Find problem drives.
  • isi quota create --path=$PATH --directory --snaps=yes --include-overhead --accounting
  • cd /ifs; touch LINTEST; isi get -DD LINTEST | grep LIN; rm LINTEST: Find the current maximum LIN.

Thursday, April 26 2012

Skyrim Tips

  • Use the Wait button to detect nearby enemies -- if you can wait, the area is clear.
  • Do not improve unimportant skills. Enemy toughness is based on your overall level. So, for instance, if you raise your Alchemy from 0 to 100, your overall level might go up and all enemies might as well. In terms of combat, it's good to have the lowest overall level but the strongest combat skills you're actively using, along with whatever auxiliary skills you prefer (such as smithing & enchanting for your gear). On the other hand, loot is also leveled...
  • Many companions (those who start with bows) won't use superior bows in combat, although they will use hand-to-hand weapons & armor you provide. They will also use better arrows; give your companion one of your best arrow -- they never use it up, and you can police them off dead enemies. Companions often tend to choose the wrong weapon or armor -- you might need to take away one piece to make them reconsider.
  • Weapons matter much more for companions than armor because they generally cannot be killed.
  • Don't give your companion a staff if you have a horse (or dog?). They're sloppy and liable to start a fight by accidentally attacking your pet.
  • Most dungeons loop back to end by the entrance. Find chests (or other containers), periodically dump all the stuff you don't need soon -- I normally do this before going through a portal to another section -- and sweep back through after you have cleared the whole dungeon to get your loot.
  • To level Smithing, create iron daggers. To level Enchanting, enchant them with Banish (this is how I use up all my Petty Soul Gems). Then sell them for all the money you'll ever need.
  • Pick a type of weapon and a type of armor and specialize. I picked Archery and Heavy Armor, although if I had known that Light Armor can (eventually) provide the maximum Armor Rating I might have picked that instead.
  • Find a chest you must steal from that's easy to get to. You can stash stolen goods in it and have your companion steal them to launder the items, removing the Stolen flag.
  • If you get the wrong soul in a gem, drop it on the ground to empty it.
  • The Unofficial Elder Scrolls Wiki seems to be the best reference.
  • On Xbox 360 scrolling gets faster briefly if you use both the left thumbstick and the D-pad to move up or down.
  • Don't bother exploring and clearing dungeons unless you're on a mission -- if you do, you will probably have to run through it again as part of a quest later.


  • Some quests are broken. I cannot complete the Companions questline because I need to kill someone who I already killed; I cannot start the Bards questline because I already spoke to the head of the College and now he won't say his line.
  • The inventory system is broken. Normally when you remove something the next item down pops up under the cursor, but sometimes the next item above is selected instead. This is carried over from Fallout.
  • The Stolen system is broken. It seems like items of the same type (and graphic) are supposed to stack, with Stolen items (marked "Stolen" in your inventory, but colored red instead in containers) on top. So you should always be able to grab the stolen items and leave a stack of un-hot items behind. But the ordering doesn't work right. It would be much better if Stolen and non-Stolen items didn't stack together.
  • The 'Cleared' flag on a dungeon means it was cleared -- it does not mean it's still clear.

Tuesday, January 31 2012

iOS Multilauncher

Several times a day on my iPhone, I tap: Home, Mail, Home, Tweetbot, Home, Reeder, Instapaper. Then I repeat the cycle, this time waiting for each to finish fetching updates and then reading what they fetched (and clipping to Instapaper from Tweetbot & Reeder) before finally ending up in Instapaper.

I do this in the morning and when emerging from the subway; I perform a variation before entering the subway and giving up 3G, and often when exiting WiFi coverage. On the iPad I have a similar routine, swapping Twitter & Flipboard for Tweetbot.

This is annoying! I am wasting my time on stupid button-mashing with substantial built-in delays -- especially on 3G.

Apple's highly effective sandbox security model, combined with iOS's much-appreciated simplicity (specifically Apple's restrictions on background tasks), mean I cannot use a cron type program to update these apps on a schedule (as I used to do on my Treo 650 for Plucker), and there is no 'wrapper' program which can tell them all to update.

Fortunately there is a way! Apple supports URL schemes for inter-application communication, and these apps use such URLs to communicate with each other -- mostly to clip articles, tweet, and send email. There is even a specification for bidirectional communication: x-callback-url.

Imagine an app named Multilauncher, designed to drive other apps in series via URL schemes. It could register the URL scheme 'multilauncher://', and come with a list of known and supported applications -- each with its own URL scheme, and preferably 'linkback' support.

So on my iPhone, in Multilauncher's settings, I could configure:

  1. Mail
  2. Tweetbot
  3. Reeder
  4. Instapaper

On my iPad, I might configure:

  1. Mail
  2. Twitter
  3. Reeder
  4. Flipboard
  5. Instapaper

I would tap Multilauncher, which would then invoke each of the specified apps. For non-callback apps (such as Mail.app) I would hit Home once they had a chance to update, and return to Multilauncher manually. Even better, though, cooperating apps could automatically relaunch Multilauncher, enabling a string of application launches & updates without manual intervention. On my iPhone, Multilauncher might launch URLs such as:

  1. mailto:
  2. tweetbot://x-callback-url/return?x-source=multilauncher://tweetbot
  3. reeder://x-callback-url/return?x-source=multilauncher://reeder
  4. instapaper://x-callback-url/return?x-source=multilauncher://instapaper

Of course Multilauncher would record what it launched last, so it could resume the sequence even without help from other apps.

Extra Credit

I don't know believe Apple currently supports launching apps from push messages or notifications, but I would be happy to subscribe to a (cheap) service to send my iPhone & iPad push messages in the morning, triggering Multilauncher so those apps could all have an opportunity to update themselves. Repeat shortly before the end of my workday. Set a timer and send a push message 6 hours after the last update. In unattended mode skip over uncooperative apps like Mail to avoid getting stuck in the sequence...

Wouldn't it be neat if the Reminder app knew when we passed in and out of 3G and WiFi coverage? It could update every time I enter WiFi, and when regaining coverage after 30+ minutes off the air. Update at the subway exits I routinely use.

Somebody please build it!

I am aware of App Switcher but it is designed as an interactive launcher -- not what I want.

Friday, January 13 2012

Big Balls' premiere & Les Sans Culottes at The Rock Shop

Thursday night at The Rock Shop, Out of Order opened. I didn't really see their set.

Then Big Balls, an AC/DC cover band, played their premier show -- AC/DC's entire first album High Voltage and the first song from their second: "Dirty Deeds Done Dirt Cheap".

Finally Les Sans Culottes played an excellent set.

  1. Monsters
  2. Allô Allô
  3. Chaussures
  4. Gangsteur d’Amour
  5. Boots
  6. Je Suis Content
  7. F.U.C. Something
  8. Les Cactus
  9. Jour du Vélo
  10. Magic Bag
  11. Téléphone Douche
  12. Hypocrite Lecteur

Encores: Ecole du Merde & Shuba Duba Luba

Tuesday, September 20 2011

Molly Does Not Approve, Pianos, September 20, 2011

Molly Mae and the latest iteration of her disapproval played Pianos again, with new guitarist Peter Goodrich and guest Chevonne on vocals & keyboards. The Micks opened. Interestingly I hear Peter's guitar more clearly in the video.

Photos (including The Micks)


  1. Good Lookin'
  2. I Wanna Have Your Baby
  3. Social Worker
  4. Pinup Girl
  5. Cancao Brasileira
  6. Garter Belt, by Chevonne
  7. Stop Stealing My Shit

Thursday, September 8 2011

Upright Piano Brigade 4x2: Marc Peloquin & David Del Tredici at Barbes

Marc Peloquin played at Barbes, with David Del Tredici -- half pieces by David, and much of it four hands. Dennis Tobenski was also present.

72 photos

  1. Marc & David: Dolly Suite, Op. 56, Gabriel Fauré
  2. Marc: Novelette No. 1, Robert Schumann
  3. Marc: Song Suite: Four Songs, Ned Rorem (transcribed for piano solo by Marc)
  4. Marc: Growl, Dennis Tobenski
  5. Marc & David: Suite, Op. 11, David Del Tredici
  6. David: Fantasy on a Cherished Name (In Memoriam, Andrew Imbrie), David Del Tredici
  7. Marc: Farewell, R.W., David Del Tredici
  8. Marc & David: Carioca Boy- Tango, David Del Tredici

Thursday, August 18 2011

Cluster job distribution & general Isilon status

Users of our Isilon clusters need basic status information, so every 10 minutes our clusters run status.sh per /etc/mcp/templates/crontab. This provides a variety of useful information to users with access to the Isilon shared filesystem, and no need to provide shell access to the cluster nodes or remember the command syntax.

We now need to run some large/slow jobs, so I wanted a list of nodes in least-busy order. Obviously Isilon tracks this so SmartConnect can send connections to the least loaded node when using the "CPU Usage" connection policy, but it's not available to user scripts. The pipeline to provide a list of nodes sorted by lowest utilization to highest is applicable to all clusters, though -- just swap in the appropriate local cluster-wide execution command for isi_for_array.


# Record basic cluster health information


isi status                   > $PREFIX/status.log
isi status -q -d             > $PREFIX/pool.log
isi job status -v            > $PREFIX/job.log
isi quota list               > $PREFIX/quota.log
isi quota list|grep -v :|grep -v default- > $PREFIX/quota-short.log
isi snapshot list -l         > $PREFIX/snapshot.log
isi snapshot usage | tail -1 > $PREFIX/snapshot-total.log
isi sync policy report | tail> $PREFIX/synciq.log
isi_for_array -s uptime      > $PREFIX/uptime.log
isi_for_array uptime | tr -d :, | awk '{print $12, $1}' | sort -n | awk '{print $2}' > $PREFIX/ordered-nodes.txt

Friday, August 12 2011

Brag 2011, with John Bianchi

My old boss from the National Audubon Society, John Bianchi, played ukelele (actually banjo/uke -- someone else played traditional uke) at the Brag vaudeville show. It was fun, and surprisingly gender-bendy. Charles Goonan was the MC. He was funny, but spent too much time onstage -- considering the final acts had to cut to make time. Alas, iMovie ate much of my footage.

Brag 2011


Video of John

  1. The Sheik of Avenue B
  2. John Bianchi: With My Little Ukulele in My Hand

The Bill

  1. Amazing Amy (contortions)
  2. Stone and Stone (standup)
  3. Vic Ruggiero (guitar)
  4. Elena Giordano (dance)
  5. Rosie Rebel
  6. Leiybya Rogers (guitar)
  7. D'yan Forest (ukelele)
  8. Richard (Rosie)
  9. Danny Cohen (standup)
  10. Trixie (burlesque)
  11. John Bianchi (banjo ukelele)
  12. Rufus Khan (standup)
  13. Vic Ruggiero & the Slackers (with everyone)

Wednesday, July 27 2011

OpenSSH is smart about cluster hostkeys

Normally, the first time you ssh to a new server, OpenSSH asks for permission to store the server's hostname (and IP) along with its unique ssh hostkey in ~/.ssh/known_hosts. Then if the hostkey ever changes, either because the machine was rebuilt or because you're connected to a different machine (as would be the case if someone intercepted your connection, for instance...), OpenSSH complains loudly that something is hinky:

pepper@teriyaki:~$ ssh cluster uname -a
The DSA host key for cluster has changed,
and the key for the corresponding IP address
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the DSA host key has just been changed.
The fingerprint for the DSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /Users/pepper/.ssh/known_hosts to get rid of this message.
Offending key in /Users/pepper/.ssh/known_hosts:81
DSA host key for cluster has changed and you have requested strict checking.
Host key verification failed.

This is a nuisance with high-availability (HA) clusters, where multiple nodes may share a single hostname and IP. The first time you connect to a shared IP everything works and you store the hostkey for whichever node accepted your connection. Then it may continue to work for a long time, if you keep connecting to the same node. But when you get a different node at that IP, OpenSSH detects it's a different machine (hostkey), and either the connection fails (if it's non-interactive) or you get the scary warning (if it's interactive). To avoid this, the convention is to ssh directly into individual nodes for administration.

But some of our sequencers use rsync-over-ssh to export data to our Isilon storage clusters, so we had a problem. If we configured them to connect to the VIP (like NFS clients), things would break when they connected to different nodes. But if we configured them to connect to individual nodes, we'd lose failover -- if any Isilon node went down, all of 'its' clients would stop transferring data until it came back up.

I briefly considered synchronizing the ssh hostkeys between nodes, to avoid the hostkey errors, but this is poor security -- if each node has the same hostkey, it's easy for any node to eavesdrop on connections to all its peers with the same hostkey, and changing keys is disruptive.

Fortunately the OpenSSH developers are way ahead of me. If the hostkey is already on file as valid for a known host -- even if there are other conflicting keys on file for the same host -- OpenSSH accepts it.

To set this up, just ssh to each node, then append the cluster hostname and IPs to their entries in ~/.ssh/known_hosts or /etc/ssh/ssh_known_hosts.

cluster-1,,cluster,,,, ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-2,,cluster,,,, ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-3,,cluster,,,, ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-4,,cluster,,,, ssh-dss AAAAB3NzaC1kc3MAAACBA...

- page 1 of 18