Extra Pepperoni

To content | To menu | To search

Friday, April 27 2012

Isilon Notes, 2012 Edition

By Chris Pepper on Friday, April 27 2012, 14:41 - computers

General

  • Isilon provides templates for Nagios, which you should use. Unfortunately Nagios cannot distinguish serious problems (failed disk) from trivia (quota violations & bogus warnings).

Hardware

  • Isilon's current units are either 2U (12-bay 200 series) or 4U (36-bay 400 series).
  • The new NL400-108 nodes are similar enough to the older 108NL nodes that they pool together. The 108NLs are dual-socket 16gb nodes based on the 72000x chassis, which is an upgrade from the 36000x chassis. This makes them much faster than the older single-core 36NLs & 72NLs.
  • As of OneFS v6.0(?), Isilon nodes no longer use the VGA keyboard & mouse console. Instead they use the serial port exclusively as console, although the VGA port does display some booting messages. In 2011, a USB connection to a KVM made a node reboot until we disconnected USB.
  • Every node is assigned a device ID when it is joined to the cluster. All alerts are tagged with the device ID of the node reporting the event. Device IDs are never reused, so if a chassis fails and is swapped out, the replacement will get a new device ID, but the old node's hostname. If this happens to you, you may want to use isi config (with advice from Isilon Support) to change the hostname to match the device ID. With a large or dynamic cluster it might just be better to ignore device IDs and let the node names run in a contiguous sequence.

Jobs

  • Isilon's job engine is problematic. Only one job runs at a time, and jobs are not efficiently parallelized.
  • MultiScan combines Collect and AutoBalance jobs.
  • During the Mark phase of Collect (or MultiScan), with snapshots enabled, delete is slow and can cause NFS timeouts.
  • It is fine for non-disruptive jobs to run in the background for long periods, and it is understandable for high-priority jobs to briefly impact the cluster, but there are too many jobs (SmartPools, AutoBalance, Collect, MultiScan) which have a substantial impact on performance for long periods.
  • There are enough long-running jobs that it's easy to get into a cycle where as soon as one finishes another resumes, meaning a job is always running and the cluster never actually catches up. It took months for us to get this all sorted out so the jobs run safely in the background and don't interfere badly.
  • When a drive does not respond quickly, Isilon logs a 'stall' in /var/log/messages. Stalls trigger "group changes", which can trigger jobs. Group changes also disrupt jobs including MultiScan, AutoBalance, & MediaScan from completing. The workaround is to tune /etc/mcp/override/sysctl.conf per Isilon Support.
  • The default job priorities were dysfunctional for us. We had to alter priorites for AutoBalance, SnapshotDelete, SmartPools, and QuotaScan, and frequency for at least SmartPools. This improved somewhat in v6.5.
  • To tweak job priority, do not redefine an existing priority. This caused problems as the change cascaded to other jobs. Define a new priority instead.

Batch Jobs

  • /etc/mcp/templates/crontab is a cluster-wide crontab; field #6 is username.

Support & Diagnostics

  • By default, Isilon's main diagnostic command, isi_gather_info, builds a tarball of configuration and logs and uploads it to EMC. This took over 15 minutes on our clusters. To make this quicker, change "Gather mode" to Incremental under Help:Diagnostics:Settings.
  • Isilon does not actually maintain an HTTP upload server, so uncheck HTTP upload to avoid a wasted timeout.
  • When a node crashes it logs a core in /var/crash, which can fill up. Upload the log with 'isi_gather_info -s "isi_hw_status -i" -f /var/crash' on the affected node before deleting it.

Network & DNS

  • Isilon is "not compatible" with firewalls, so client firewalls must be configured to allow all TCP & UDP ports from Isilon nodes & pools back to NFS clients (and currently SNMP consoles).
  • Specifically, there is a bug where SNMP responses come from the node's primary IP. iptables on our Nagios console dropped responses which came from a different IP than Nagios queried.
  • To use SmartConnect you must delegate the Isilon domain names to the SmartConnect resolver on the cluster. We were unable to use DNS forwarding in BIND with this delegation active.

NFS

  • By default Isilon exports a shared large /ifs filesystem from all nodes. They suggest mounting with /etc/fstab options rw,nfsvers=3,rsize=131072,wsize=524288.

CIFS

  • Migrating an IP to another node disconnects CIFS clients of that IP.
  • CIFS clients should use their own static SmartConnect pools rather than connecting to dynamic SmartConnect pools (for NFS clients).

Load Balancing

  • Rather than real-time load balancing, Isilon handles load-balancing through its built-in DNS server (SmartConnect: Basic or Advanced). Because this happens at connection time, the cluster cannot manage load between clients which are already connected, except via "isi networks --sc-rebalance-all", which shuffles server-side IPs in to even out load. Unfortunately OneFS (as of v6.5) does not track utilization statistics for network connections, so it cannot intelligently determine how much traffic each IP represents. This means only Round Robin and Connection Count are suitable for "IP failover policy" (rebalancing) -- "Network Throughput" & "CPU Usage" don't work.
  • High availability is handled by reassigning IPs to different nodes in case of failure. For NFS this is seamless, but for CIFS this causes client disconnection. As a result CIFS clients must connect to static pools, and "isi networks --sc-rebalance-all" should never be run on clusters with CIFS clients (there is apparently a corresponding command to rebalance a single pool, suitable for manual use on each dynamic pool).

Quotas

  • Some of the advantage of the single filesystem is lost because it is impossible to move files from one quota under another. This forces us to copy (rsync) and then delete as if each quota were its own mount point.
  • For user quota reporting, each user should have an account (perhaps via LDAP or AD) on the cluster.
  • For user quota notifications, each user must have an email mapping (we created aliases to route machine account quota notifications to the right users).

Bugs

  • The user Enable checkbox disables all login access (but preserves UID mappings for quota reports). Unchecking it blocks both ssh and CIFS/SMB access and clears the user password.
  • You cannot create a user with a home directory that exists (even with --force). Workaround: move the directory aside before creating the user, or create with a bogus homedirectory (which can only be used once) and use "isi auth local user modify" to fix after creation.
  • Don't use more than 8 SyncIQ policies (I don't know if this bug has been fixed).
  • Gateways and priorities are not clear, but if there are 2 gateways with the same priority the cluster can get confused and misbehave. The primary gateway should have the lowest priority number (1).
  • We heard one report that advisory quotas on a SyncIQ target cluster caused SyncIQ errors.
  • If you configure two gateways with the same priority, the cluster can get confused and misbehave.
  • In at least one case, advisory quotas on a SyncIQ target disrupted SyncIQ.
  • The Virtual Hot Spare feature appears to reserve twice as many drives as are specified in the UI, and they do not work as described.

Support

  • Support is very slow. SLAs apparently only apply to parts delivery -- our 4-hour service does not prevent Isilon from saying they will answer questions in a few days.
  • Support is constantly backlogged. Callback times are rarely made and cases are often not followed up unless we call in to prod Support.
  • My process for opening a case looks like this:
    1. Run uname -a; isi_hw_status -i; isi_gather_info.
    2. Paste output from first 2 commands and gather filename into email message.
    3. Describe problem and send email to support@.
    4. A while later we get a confirmation email with a case number.
    5. A day or two later I get tired of waiting and phone Isilon support.
    6. I punch in my case number from the acknowledgement.
    7. I get a phone rep and repeat the case number.
    8. The phone rep transfers me to a level 1 support rep, who as a rule cannot answer my question.
    9. The L1 rep tries to reach an L2 rep to address my question. They are often unable to reach anyone(!!!), and promise a callback as soon as they find an L2 rep.
    10. As a rule, I do not receive a callback.
    11. Eventually I give up on waiting and call in again.
    12. I describe my problem a third time.
    13. The L1 tech goes off to find an answer.
    14. I may have to call back in and prod L1 multiple times (there is no way for me to reach L2 directly).
    15. Eventually I get an answer. This process often takes over a week.
  • Support provides misinformation too often. Most often this is simple ignorance or confusion, but it appears to be EMC policy to deny that any problem affects multiple sites.

Commands

For manual pages, use an underscore (e.g., man isi_statistics). The command line is much more complete than the web interface but not completely documented. Isilon uses zsh with customized tab completion. When opening a new case include output from "uname -a" & "isi_hw_status -i", and run isi_gather_info.

  • isi_for_array -s: Execute a command on all nodes in in order.
  • isi_hw_status -i: Node model & serial number -- include this with every new case.
  • isi status: Node & job status. -n# for particular node, -q to skip job status, -d for SmartPool utilization; we use isi status -qd more often.
  • isi statistics pstat --top & isi statistics protocol --protocol=nfs --nodes=all --top --long --orderby=Ops
  • isi networks
  • isi alerts list -A -w: Review all alerts.
  • isi alerts cancel all: Clear existing alerts, including the throttled critical errors message. Better than the '''Quiet''' command, which can suppress future errors as well.
  • isi networks --sc-rebalance-all: Redistribute SmartConnect IPs to rebalance load. Not suitable for clusters with CIFS shares.
  • du -A: Size, excluding protection overhead, from an Isilon node.
  • du --apparent-size: Size, excluding protection overhead, from a Linux client.
  • isi devices: List disks with serial numbers.
  • isi snapshot list --schedule
  • isi snapshot usage | grep -v '0.0'
  • isi quota list --show-with-no-overhead | isi quota list --show-with-overhead | isi quota list --recurse-path=/ifs/nl --directory
  • isi quota modify --directory --path=/ifs/nl --reset-notify-state
  • isi job pause MultiScan / isi job resume MultiScan
  • isi job config --path jobs.types.filescan.enabled=False: Disable MultiScan.
  • isi_change_list (unsupported): List changes between snapshots.
  • sysctl -n hw.physmem: Check RAM.
  • isi device -a smartfail -d 1:bay6 / isi devices -a stopfail -d 1:bay6 (stopfail is not normally appropriate)
  • isi devices -a add -d 12:10: Use new disk in node 12, bay 10.
  • date; i=0; while [ $i -lt 36 ]; do isi statistics query --nodes=1-4 --stats=node.disk.xfers.rate.$i; i=$[$i+1]; done # Report disk IOPS(?) for all disks in nodes 1-4 -- 85-120 is apparently normal for SATA drives.
  • isi networks modify pool --name *$NETWORK*:*$POOL* --sc-suspend-node *$NODE*: Prevent $POOL from offering $NODE for new connections, without interfering with active connections. --sc-resume-node to undo.
  • isi_lcd_d restart: Reset LEDs.
  • isi smb config global modify --access-based-share-enum=true: Restrict SMB shares to authorized users (global version); isi smb config global list | grep access-based: verify (KB #2837)
  • ifa isi devices | grep -v HEALTHY: Find problem drives.
  • isi quota create --path=$PATH --directory --snaps=yes --include-overhead --accounting
  • cd /ifs; touch LINTEST; isi get -DD LINTEST | grep LIN; rm LINTEST: Find the current maximum LIN.

no comment

Thursday, April 26 2012

Skyrim Tips

By Chris Pepper on Thursday, April 26 2012, 20:53 - computers

  • Use the Wait button to detect nearby enemies -- if you can wait, the area is clear.
  • Do not improve unimportant skills. Enemy toughness is based on your overall level. So, for instance, if you raise your Alchemy from 0 to 100, your overall level might go up and all enemies might as well. In terms of combat, it's good to have the lowest overall level but the strongest combat skills you're actively using, along with whatever auxiliary skills you prefer (such as smithing & enchanting for your gear). On the other hand, loot is also leveled...
  • Many companions (those who start with bows) won't use superior bows in combat, although they will use hand-to-hand weapons & armor you provide. They will also use better arrows; give your companion one of your best arrow -- they never use it up, and you can police them off dead enemies. Companions often tend to choose the wrong weapon or armor -- you might need to take away one piece to make them reconsider.
  • Weapons matter much more for companions than armor because they generally cannot be killed.
  • Don't give your companion a staff if you have a horse (or dog?). They're sloppy and liable to start a fight by accidentally attacking your pet.
  • Most dungeons loop back to end by the entrance. Find chests (or other containers), periodically dump all the stuff you don't need soon -- I normally do this before going through a portal to another section -- and sweep back through after you have cleared the whole dungeon to get your loot.
  • To level Smithing, create iron daggers. To level Enchanting, enchant them with Banish (this is how I use up all my Petty Soul Gems). Then sell them for all the money you'll ever need.
  • Pick a type of weapon and a type of armor and specialize. I picked Archery and Heavy Armor, although if I had known that Light Armor can provide the maximum Armor Rating I might have picked that instead.
  • The inventory system is broken. Normally when you remove something the next item down pops up under the cursor, but sometimes the next item above is selected instead. This is carried over from Fallout.
  • The Stolen system is broken. It seems like items of the same type (and graphic) are supposed to stack, with Stolen items (marked "Stolen" in your inventory, but colored red instead in containers) on top. So you should always be able to grab the stolen items and leave a stack of un-hot items behind. But the ordering doesn't work right. It would be much better if Stolen and non-Stolen items didn't stack together.
  • Find a chest you must steal from that's easy to get to. You can stash stolen goods in it and have your companion steal them to launder the items, removing the Stolen flag.
  • If you get the wrong soul in a gem, drop it on the ground to empty it.
  • The Unofficial Elder Scrolls Wiki seems to be the best reference.
  • On Xbox 360 scrolling gets faster briefly if you use both the left thumbstick and the D-pad to move up or down.

no comment

Tuesday, January 31 2012

iOS Multilauncher

By Chris Pepper on Tuesday, January 31 2012, 15:00 - iPod / iPhone

Several times a day on my iPhone, I tap: Home, Mail, Home, Tweetbot, Home, Reeder, Instapaper. Then I repeat the cycle, this time waiting for each to finish fetching updates and then reading what they fetched (and clipping to Instapaper from Tweetbot & Reeder) before finally ending up in Instapaper.

I do this in the morning and when emerging from the subway; I perform a variation before entering the subway and giving up 3G, and often when exiting WiFi coverage. On the iPad I have a similar routine, swapping Twitter & Flipboard for Tweetbot.

This is annoying! I am wasting my time on stupid button-mashing with substantial built-in delays -- especially on 3G.

Apple's highly effective sandbox security model, combined with iOS's much-appreciated simplicity (specifically Apple's restrictions on background tasks), mean I cannot use a cron type program to update these apps on a schedule (as I used to do on my Treo 650 for Plucker), and there is no 'wrapper' program which can tell them all to update.

Fortunately there is a way! Apple supports URL schemes for inter-application communication, and these apps use such URLs to communicate with each other -- mostly to clip articles, tweet, and send email. There is even a specification for bidirectional communication: x-callback-url.

Imagine an app named Multilauncher, designed to drive other apps in series via URL schemes. It could register the URL scheme 'multilauncher://', and come with a list of known and supported applications -- each with its own URL scheme, and preferably 'linkback' support.

So on my iPhone, in Multilauncher's settings, I could configure:

  1. Mail
  2. Tweetbot
  3. Reeder
  4. Instapaper

On my iPad, I might configure:

  1. Mail
  2. Twitter
  3. Reeder
  4. Flipboard
  5. Instapaper

I would tap Multilauncher, which would then invoke each of the specified apps. For non-callback apps (such as Mail.app) I would hit Home once they had a chance to update, and return to Multilauncher manually. Even better, though, cooperating apps could automatically relaunch Multilauncher, enabling a string of application launches & updates without manual intervention. On my iPhone, Multilauncher might launch URLs such as:

  1. mailto:
  2. tweetbot://x-callback-url/return?x-source=multilauncher://tweetbot
  3. reeder://x-callback-url/return?x-source=multilauncher://reeder
  4. instapaper://x-callback-url/return?x-source=multilauncher://instapaper

Of course Multilauncher would record what it launched last, so it could resume the sequence even without help from other apps.

Extra Credit

I don't know believe Apple currently supports launching apps from push messages or notifications, but I would be happy to subscribe to a (cheap) service to send my iPhone & iPad push messages in the morning, triggering Multilauncher so those apps could all have an opportunity to update themselves. Repeat shortly before the end of my workday. Set a timer and send a push message 6 hours after the last update. In unattended mode skip over uncooperative apps like Mail to avoid getting stuck in the sequence...

Wouldn't it be neat if the Reminder app knew when we passed in and out of 3G and WiFi coverage? It could update every time I enter WiFi, and when regaining coverage after 30+ minutes off the air. Update at the subway exits I routinely use.

Somebody please build it!

I am aware of App Switcher but it is designed as an interactive launcher -- not what I want.

no comment

Friday, January 13 2012

Big Balls' premiere & Les Sans Culottes at The Rock Shop

By Chris Pepper on Friday, January 13 2012, 00:15 - music

Thursday night at The Rock Shop, Out of Order opened. I didn't really see their set.

Then Big Balls, an AC/DC cover band, played their premier show -- AC/DC's entire first album High Voltage and the first song from their second: "Dirty Deeds Done Dirt Cheap".

Finally Les Sans Culottes played an excellent set.

  1. Monsters
  2. Allô Allô
  3. Chaussures
  4. Gangsteur d’Amour
  5. Boots
  6. Je Suis Content
  7. F.U.C. Something
  8. Les Cactus
  9. Jour du Vélo
  10. Magic Bag
  11. Téléphone Douche
  12. Hypocrite Lecteur

Encores: Ecole du Merde & Shuba Duba Luba

no comment

Tuesday, September 20 2011

Molly Does Not Approve, Pianos, September 20, 2011

By Chris Pepper on Tuesday, September 20 2011, 22:00 - music

Molly Mae and the latest iteration of her disapproval played Pianos again, with new guitarist Peter Goodrich and guest Chevonne on vocals & keyboards. The Micks opened. Interestingly I hear Peter's guitar more clearly in the video.

Photos (including The Micks)

Videos

  1. Good Lookin'
  2. I Wanna Have Your Baby
  3. Social Worker
  4. Pinup Girl
  5. Cancao Brasileira
  6. Garter Belt, by Chevonne
  7. Stop Stealing My Shit

one comment

Thursday, September 8 2011

Upright Piano Brigade 4x2: Marc Peloquin & David Del Tredici at Barbes

By Chris Pepper on Thursday, September 8 2011, 21:00 - music

Marc Peloquin played at Barbes, with David Del Tredici -- half pieces by David, and much of it four hands. Dennis Tobenski was also present.

72 photos

  1. Marc & David: Dolly Suite, Op. 56, Gabriel Fauré
  2. Marc: Novelette No. 1, Robert Schumann
  3. Marc: Song Suite: Four Songs, Ned Rorem (transcribed for piano solo by Marc)
  4. Marc: Growl, Dennis Tobenski
  5. Marc & David: Suite, Op. 11, David Del Tredici
  6. David: Fantasy on a Cherished Name (In Memoriam, Andrew Imbrie), David Del Tredici
  7. Marc: Farewell, R.W., David Del Tredici
  8. Marc & David: Carioca Boy- Tango, David Del Tredici

no comment

Thursday, August 18 2011

Cluster job distribution & general Isilon status

By Chris Pepper on Thursday, August 18 2011, 12:34 - computers

Users of our Isilon clusters need basic status information, so every 10 minutes our clusters run status.sh per /etc/mcp/templates/crontab. This provides a variety of useful information to users with access to the Isilon shared filesystem, and no need to provide shell access to the cluster nodes or remember the command syntax.

We now need to run some large/slow jobs, so I wanted a list of nodes in least-busy order. Obviously Isilon tracks this so SmartConnect can send connections to the least loaded node when using the "CPU Usage" connection policy, but it's not available to user scripts. The pipeline to provide a list of nodes sorted by lowest utilization to highest is applicable to all clusters, though -- just swap in the appropriate local cluster-wide execution command for isi_for_array.

status.sh

#!/bin/sh
# Record basic cluster health information

PREFIX=/ifs/x/common/cluster/status

isi status                   > $PREFIX/status.log
isi status -q -d             > $PREFIX/pool.log
isi job status -v            > $PREFIX/job.log
isi quota list               > $PREFIX/quota.log
isi quota list|grep -v :|grep -v default- > $PREFIX/quota-short.log
isi snapshot list -l         > $PREFIX/snapshot.log
isi snapshot usage | tail -1 > $PREFIX/snapshot-total.log
isi sync policy report | tail> $PREFIX/synciq.log
isi_for_array -s uptime      > $PREFIX/uptime.log
isi_for_array uptime | tr -d :, | awk '{print $12, $1}' | sort -n | awk '{print $2}' > $PREFIX/ordered-nodes.txt

no comment

Friday, August 12 2011

Brag 2011, with John Bianchi

By Chris Pepper on Friday, August 12 2011, 23:00 - pictures

My old boss from the National Audubon Society, John Bianchi, played ukelele (actually banjo/uke -- someone else played traditional uke) at the Brag vaudeville show. It was fun, and surprisingly gender-bendy. Charles Goonan was the MC. He was funny, but spent too much time onstage -- considering the final acts had to cut to make time. Alas, iMovie ate much of my footage.

Brag 2011

Photos

Video of John

  1. The Sheik of Avenue B
  2. John Bianchi: With My Little Ukulele in My Hand

The Bill

  1. Amazing Amy (contortions)
  2. Stone and Stone (standup)
  3. Vic Ruggiero (guitar)
  4. Elena Giordano (dance)
  5. Rosie Rebel
  6. Leiybya Rogers (guitar)
  7. D'yan Forest (ukelele)
  8. Richard (Rosie)
  9. Danny Cohen (standup)
  10. Trixie (burlesque)
  11. John Bianchi (banjo ukelele)
  12. Rufus Khan (standup)
  13. Vic Ruggiero & the Slackers (with everyone)

no comment

Wednesday, July 27 2011

OpenSSH is smart about cluster hostkeys

By Chris Pepper on Wednesday, July 27 2011, 11:29 - ssh

Normally, the first time you ssh to a new server, OpenSSH asks for permission to store the server's hostname (and IP) along with its unique ssh hostkey in ~/.ssh/known_hosts. Then if the hostkey ever changes, either because the machine was rebuilt or because you're connected to a different machine (as would be the case if someone intercepted your connection, for instance...), OpenSSH complains loudly that something is hinky:

pepper@teriyaki:~$ ssh cluster uname -a
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The DSA host key for cluster has changed,
and the key for the corresponding IP address 10.0.10.124
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the DSA host key has just been changed.
The fingerprint for the DSA key sent by the remote host is
f7:b0:d4:11:2c:6c:ec:be:96:f0:88:71:d9:26:20:0c.
Please contact your system administrator.
Add correct host key in /Users/pepper/.ssh/known_hosts to get rid of this message.
Offending key in /Users/pepper/.ssh/known_hosts:81
DSA host key for cluster has changed and you have requested strict checking.
Host key verification failed.

This is a nuisance with high-availability (HA) clusters, where multiple nodes may share a single hostname and IP. The first time you connect to a shared IP everything works and you store the hostkey for whichever node accepted your connection. Then it may continue to work for a long time, if you keep connecting to the same node. But when you get a different node at that IP, OpenSSH detects it's a different machine (hostkey), and either the connection fails (if it's non-interactive) or you get the scary warning (if it's interactive). To avoid this, the convention is to ssh directly into individual nodes for administration.

But some of our sequencers use rsync-over-ssh to export data to our Isilon storage clusters, so we had a problem. If we configured them to connect to the VIP (like NFS clients), things would break when they connected to different nodes. But if we configured them to connect to individual nodes, we'd lose failover -- if any Isilon node went down, all of 'its' clients would stop transferring data until it came back up.

I briefly considered synchronizing the ssh hostkeys between nodes, to avoid the hostkey errors, but this is poor security -- if each node has the same hostkey, it's easy for any node to eavesdrop on connections to all its peers with the same hostkey, and changing keys is disruptive.

Fortunately the OpenSSH developers are way ahead of me. If the hostkey is already on file as valid for a known host -- even if there are other conflicting keys on file for the same host -- OpenSSH accepts it.

To set this up, just ssh to each node, then append the cluster hostname and IPs to their entries in ~/.ssh/known_hosts or /etc/ssh/ssh_known_hosts.

cluster-1,10.0.10.101,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-2,10.0.10.102,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-3,10.0.10.103,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...
cluster-4,10.0.10.104,cluster,10.0.10.121,10.0.10.122,10.0.10.123,10.0.10.124 ssh-dss AAAAB3NzaC1kc3MAAACBA...

no comment

Sunday, July 10 2011

4th St NiteOwls, July 1 2011, Barbes

By Chris Pepper on Sunday, July 10 2011, 21:30 - music

The NiteOwls played Barbes. Unfortunately I missed the beginning.

  • Stayin' Alive
  • It's No Fun
  • Crosspatch
  • When Day Is Done
  • Crazy about My Baby and My Baby Is Crazy about Me
  • Viper
  • The Porter's Love Song to a Chambermaid
  • Wish I Were Twins
  • Two Sleepy People
  • Busy Bee

one comment

Saturday, June 18 2011

Mermaid Parade 2011

By Chris Pepper on Saturday, June 18 2011, 18:00 - pictures

Bjorn and I met up at the Mermaid Parade this year. We met a couple of Bjorn's friends, who met some more of their friends, who met and made more friends -- it was a social snowball. Mariah & Danielle looked great, so people kept stopping them to take their picture, or to have pictures taken with them -- including the Sea Rabbit and its human creator.

The whole thing felt very much like part of a Brooklyn continuum. We saw crazy colorful animal dude again -- Bjorn & Mariah see him around Manhattan, and I saw both last week and last year at Brooklyn Pride. And the lime green Charger reappeared the next day on 7th Ave, in the 7th Avenue Street Fair.

I took 941 photos (a personal record) and whittled them down to 281 at Flickr.

no comment

Tuesday, March 29 2011

AT&T iPad 3G details

By Chris Pepper on Tuesday, March 29 2011, 16:00 - iPod / iPhone

We will be taking a couple short trips this year, and now that I have a cellular capable iPad it's time to figure out how I'll have Internet access. TidBITS has covered the 'new' AT&T Personal Hotspot feature on iPhones, and the trade-offs between iPad 3G service vs. connecting through an iPhone hotspot). I don't travel much, so I prefer AT&T's proratable service over Verizon's monthly service. I got the AT&T 3G iPad, rather than the Verizon 3G model or the WiFi-only model.

iPhone: Wireless Features

I thought I would link my iPad to our AT&T family plan, so I could start and stop service with myAT&T.app, but normal iPhone service (including our family plan) is under AT&T's 'postpaid' model, where each month they bill us for minutes/texts/features actually used. The prorated 3G service is prepaid, and doesn't appear to link up with iPhone service or be manageable through myAT&T.app.

Our first trip will be a long weekend with family. In Settings.app, under Cellular Data, I enter my billing info and sign up for the 250mb/30-day plan for $14.99. Hopefully that isn't prorated out to 8.33mbytes/day! The second trip will be over 8 days so I'll sign up for the $25 2gb/30-day plan. Signing up automatically activates autorenewal, so each time I will stop autorenewal after I sign up, and terminate service manually when we get back home. Apparently you can register for AT&T 3G service even without an active contract or WiFi service -- even without a contract the iPad has access to an AT&T (or Verizon) captive network where you can just sign up -- but I don't intend to test this.

iPad: Cellular Data Account

Supposedly http://att.com/ipadlanding provides a web interface to manage iPad 3G plans, but it won't accept my (postpaid iPhone) credentials, and I don't see any reason to register again.

Had I chosen to use Personal Hotspot on the iPhone instead of iPad 3G service, I could have changed from my current "DataPro 2 GB for iPhone" service to "DataPro 4GB for iPhone" (which enables Personal Hotspot) directly in myAT&T.app. Alternatively, an AT&T rep offered to schedule my changes to 4gb/tethering and back to 2gb (no tethering) on whatever dates I prefer, over the phone.

no comment

Thursday, March 17 2011

Robin Aigner in Boozin' & Bruisin' at Jalopy

By Chris Pepper on Thursday, March 17 2011, 23:00 - music

I went to Jalopy again to see Robin Aigner. Unfortunately her site had the wrong time, so I showed up a couple hours early, came back home, and went back again! The show was a benefit for the Music Maker Relief Fund, put on by The Whistlin' Wolves (who joined most of the performances, along with David Bennet Cohen on piano).

It was funny to hear Old Plank Road performed by someone other than The Magpie, although I heard Benjoya mentioned a few times. I sat next to Sheriff Uncle Bob and recognize he was dressed for the evening from his sheriff's star (and tie with more stars), but didn't realize until he went onstage that he was a performer.

I took some photos.

Videos

one comment

Thursday, March 3 2011

Parenting in the Age of the Internet

By Chris Pepper on Thursday, March 3 2011, 08:30 - Julia

Chuck Joiner ran a panel discussion at Macworld on Parenting in the Age of the Internet -- I heard about it through Tonya, one of the panelists. They had a bunch of suggestions, several new to me.

  • Keep hard rules as few and flexible as possible. You cannot expect a child to abide by a multi-page contract which doesn't make sense to them.
  • Recognize the same behaviors in yourself.
    • Model how to use computers and the Internet safely.
    • Be aware of how much time you spend using technology instead of focused on family. This is more of an issue with older kids, but young kids are already learning about engagement and disengagement. Several panelists commented on this, and it resonated for me.
  • Pay attention to game ratings, but try the games yourself first. I am pretty good about this, although as Julia gets more engaged in more complicated games which I am uninterested in, it will be harder.
  • Limit computer/game time (we also limit TV time).
  • Use parental controls.
    • The ones built into Mac OS X are pretty good, although not very deep.
    • The ones built into Windows Live are apparently more flexible.
    • Unfortunately, when Julia was using an Ubuntu netbook, I discovered that the Linux controls are lacking and poor. Partially this is due to a philosophical issue -- Linux hackers don't want to restrict anyone's freedom. They make the valid argument that parents should supervise their kids, but failed to consider that we have other things to do, and would like parental controls to help enforce policies, so kids can use computers without parents constantly watching over their shoulders.
    • You can get much more sophisticated controls over web surfing with a (Linux) router/filter. We don't need this yet, although we might someday. Network filters also work for devices (such as iPads & smartphones) without built-in restrictions.
    • OpenDNS provides a very easy-to-use filter that blocks a lot of crud. I hadn't thought about OpenDNS for this purpose, but it's an excellent idea.
    • Blocking should not be a hard stop. Instead make it an opportunity to discuss the rules and the site that was blocked -- perhaps there is a good reason to visit it. This makes a lot of sense.
  • Email is a very useful microcosm for the Internet. It's easy to keep track of and to limit to a small trusted subset of correspondents, and provides natural & strong motivation for typing and other computer skills.
    • Gmail offers delegation. We use another service with full parental/administrative access.
  • There was considerable discussion of how kids SMS incessantly -- often to the exclusion of making voice calls. Julia does not text yet but does like talking on the phone, so this isn't a problem for us yet.
  • One panelist required their children to accept their Facebook friend requests and provide their passwords. I am uncomfortable with password sharing but it does make sense here. Requiring young kids to accept parental friend requests is an obvious prerequisite for using Facebook. Of course, a clever kid can use custom controls to limit what their parents see even if they're friends.
  • The panelists didn't talk much about privacy.
    • We're clear that Julia doesn't have any privacy from us on the Internet. Now this is easy, but it will bother her as she grows up, and we don't know where independence begins -- we'll have to figure it out as we go.
    • The panelists didn't mention this, but explaining this lack of privacy is essential -- spying on kids who expect privacy destroys trust and makes it a battle between kids and parents. And there's no way to win this battle over the long term anyway.
  • A couple people talked about the importance of computer placement & access. This is why Julia doesn't have an old iPhone (no way to keep track of usage), and why the laptops & iPad she uses stay in common areas of our apartment -- she uses the iPad where we ca monitor her.
  • Don't forget parental/family/sibling privacy -- kids need to consider who else is affected by their posts & sharing.
  • Tonya pointed out that some kids who grow up without TV find that a problem for socialization. We had not encountered or considered this.
  • You cannot expect children not to make mistakes, or to make the same decisions they (or you) would as adults. Instead try to make them aware of the dangers, and if they don't care about the real-life consequences (such as Facebook profile review & Googling as part of job hiring processes), you may have to impose more immediate consequences (family rules & punishments) which are sufficiently real to have an impact on behavior.

5 comments

Saturday, February 5 2011

Freddy's Bar, Grand Reopening, February 2011

By Chris Pepper on Saturday, February 5 2011, 02:15 - music

Freddy's Bar has reopened in a new location. The new place is larger and nicer, although they still need to sort some things out -- staff kept walking through the stage during the performances. To celebrate the occasion, Freddy's hosted 3 bands on opening night: The Magpie, Brute Force, and Les Sans Culottes. Josh and I were there to see Gavin play in both The Magpie and Les Sans Culottes.

Many folks were snapping away with DSLRs and phones, and one fellow had a serious looking video camera, on a tripod, atop a raised platform. Fortunately for us all, Freddy's had the lights on.

Photos

Magpie

  1. Biguine/Biguine
  2. Cormaine
  3. Copra Luca
  4. If You Want to Know Who We Are
  5. March Hora
  6. Old Plank Road
  7. Tania
  8. Mineola Rag
  9. Mysteriouse
  10. D Minor Bulgar

Brute Force

I didn't take any video of Brute Force.

Les Sans Culottes

  1. Magic Baguette
  2. Monsters from the Id
  3. Sur la Plage
  4. Allo Allo
  5. Je Suis Content
  6. Gangsteur*
  7. Jour du Velo
  8. Boots
  9. F.U.C. Something
  10. Chaussures
  11. Hot Bird
  12. Le Biz
  13. Sartre & the Lobster*
  14. Ou Est ou Est*
  15. SOS Elefants
  16. Telefon Douche

LSC Encore

  1. Shuba Duba Luba
  2. ???
  3. Ecole du Merde (for Virpi)

*: written by Gavin

no comment

Monday, January 31 2011

Isilon Cluster

By Chris Pepper on Monday, January 31 2011, 12:00 - computers

Our old bulk storage is Apple Xserve RAIDs. They are discontinued and service contracts are expiring, so we have been evaluating small-to-medium storage options for some time. Our more modern stuff is a mix of Solaris 10 (ZFS) on Sun X4500/X4540 chassis (48 * 1tb SATA; discontinued), and Nexsan SATABeasts (42 SATA drives, either 1tb or 2tb) attached to Linux hosts, with ext3 filesystems. We are not buying any more Sun hardware or switching to FreeBSD for ZFS, and ext4 does not yet support filesystems over 16tb. Breaking up a nice large array into a bunch of 16tb filesystems is annoying, but moving (large) directories between filesystems is really irritating.

We eventually decided on a 4-node cluster of Isilon IQ 32000X-SSD nodes. Each ISI36 chassis is a 4U (7" tall) server with 24 3.5" drive bays on the front and 12 on the back. In our 32000X-SSD models, bays #1-4 are filled with SSDs (apparently 100gb each, currently usable only for metadata) and the other 32 bays hold 1tb SATA drives, thus the name. Each of our nodes has 2 GE ports on the motherboard and a dual-port 10GE card.

Isilon's OneFS operating system is based on FreeBSD, with their proprietary filesystem and extra bits added. Their OneFS cluster file system is cache coherent: inter-node lookups are handled over an InfiniBand (DDR?) backend, so any node can serve any request; most RAM on the nodes is used as cache. Rather than traditional RAID 5 or 6, the Isilon cluster stripes data 'vertically' across nodes, so it can continue to operate despite loss of an entire node. This means an Isilon cluster must consist of at least 3 matching nodes, just like a RAID5 must consist of at least 3 disks. Unfortunately, this increases the initial purchase cost considerably, but cost per terabyte decreases as node count grows, and the incremental system administration burden per node is much better than linear.

Routine administration is managed through the web interface, although esoteric options require the command line. Isilon put real work into the Tab completion dictionaries. This is quite helpful when exploring the command line interface, but the (zsh based) completions are not complete -- neither are the --help messages nor the manual pages, unfortunately.

There are many good things about Isilon.

Pros

  • Single filesystem & namespace. This sounds minor but is essential for coping with large data sets. Folders can be arbitrarily large and all capacity is available to all users/shares, subject to quotas.
  • Cost per terabyte decreases with node count, as parity data becomes a smaller proportion of total disk capacity.
  • Aggregate performance increases with node count -- total cache increases, and number of clients per server is reduced.
  • Administration burden is fairly flat with cluster growth.
  • The FlexProtect system (based on classic RAID striping-with-parity and mirroring, but between nodes rather than within nodes/shelves) is flexible and protects against whole-node failure.
  • NFS and CIFS servers are included in the base price.
  • Isilon's web UI is reasonably simple, but exposes significant power.
  • The command line environment is quite capable, and Tab completion improves discoverability.
  • Quotas are well designed, and flexible enough to use without too much handholding for exceptions.
  • Snapshots are straightforward and very useful. They are comparable to ZFS snapshots -- much better than Linux LVM snapshots (ext3 does not support snapshots directly).
  • The nodes include NVRAM and battery backup for safe high-speed writes.
  • Nodes are robust under load. Performance degrades predictably as load climbs, and we don't have to worry about pushing so hard the cluster falls over.
  • Isilon generally handles multiple network segments with aplomb.
  • The storage nodes provide complete services -- they do not require Linux servers to front-end services, or additional high availability support.
  • The disks are hot swap, and an entire chassis can be removed for service without disrupting cluster services.
  • Because the front end is gigabit Ethernet (or 10GE), an Isilon storage cluster can serve an arbitrarily large number of clients without expensive fibre channel HBAs and switches.

And, of course, some things are less good.

Cons

  • Initial/minimum investment is high: 3 matching nodes, 2 InfiniBand switches, and licenses.
  • Several additional licenses are required for full functionality.
  • Isilon is not perfectionistic about the documentation -- in fact, the docs are incomplete.
  • Isilon is not as invested in the supporting command-line environment as I had hoped.
  • The round-robin load balancing works by delegating a subdomain to the Isilon cluster. Organizationally, this might be complicated.
  • CIFS integration requires AD access for accounts. This might also be logistically difficult.
  • Usable capacity is unpredictable and varies based on data composition.
  • There are always two different disk utilization numbers: actual data size, and including protection. This is confusing compared to classic RAID, where users only see unique data size.
  • There is no good way for users to identify which node they're connected to. This is possible but awkward for administrators to determine, but it is generally not worth going beyond the basic web charts.
  • Support can be frustrating.
    • We often get responses from many people on the same case, and rehashing the background repeatedly wastes time.
    • Some reps are very good; but some are poor, with wrong answers, pointless instructions, and a disappointing lack of knowledge about the technology and products.
    • We are frequently asked for system name & serial number, and asked to upload a status report with isi_gather_info -- even when this is all already on file.
    • Minor events trigger email asking if we need help, even when we're in the middle of scheduled testing.
  • The cluster is built of off-the-shelf parts, and the integration is not always complete. For instance, we are not alerted of problems with an InfiniBand switch, because things like a faulted PSU are not visible to the nodes and not logged.
  • Many commands truncate output to 80 columns -- even when the terminal is wider. To see full output add -w.
  • When the system is fully up, the VGA console does not show a prompt. This makes it harder to determine whether a node has booted successfully.
  • There is only one bit of administrative access control: when users log in, they either have access to the full web interface and command-line tools, or they don't. There is no read-only or 'operator' mode.
  • Running out of space (or even low on space) is apparently dangerous.
  • One suggestion was to reserve one node's worth of disks as free space, so the whole cluster can run with a dead node. In a 4-node configuration, reserving 25% of raw space for robustness (in addition to 25% for parity) would mean 50% utilization at best, which is generally not feasible. In fairness, it is rare for a storage array to even attempt to work around a whole shelf failure, but most (non-Isilon) storage shelves are simple enclosures with fewer and simpler failure modes...
  • SmartConnect is implemented as a DNS server, but it's incomplete -- it only responds to A record requests, which causes errors when programs like host attempt other queries.
  • The front panels are finicky. The controls are counterintuitive, the LED system is prone to bizarre (software) failure modes, and removing the front panel to access the disks raises an obscure but scary alert.

Notes

  • On Isilon nodes, use du -Sl to get size without protection overhead. On Linux clients, use du --apparent-size.
  • Client load balancing is normally managed via DNS round robin, with the round robin addresses automatically redistributed in case of a node failure. This is less granular and balanced than you'd get from a full load balancer, but much simpler.
  • EMC has bought Isilon. I'm not sure what the impact will be, but I am not confident this will be a good thing over the long term.
  • In BIND (named), subdomain delegation is incompatible with forwarding. Workaround: Add forwarders {}; to zone containing Isilon NS record.

Future

  • All that said, we are getting more Isilon storage -- it seems like the best fit for our requirements.

one comment

Monday, January 24 2011

Canon Vixia HF S20

By Chris Pepper on Monday, January 24 2011, 01:00 - pictures

I got a Sony video camera to take video of Julia, but an 8+ year old video camera is useless -- both the iPhone and the Canon Rebel T1i take better (and higher resolution) video, but the iPhone has no zoom and neither has high-quality audio. I have been filming and photographing friends' bands for a while now, and am getting a bit more serious about it. Unfortunately, musicians are allergic to light, so it's always an adverse environment for any kind of camera. It was time to find a video camera that can handle low light, with good/flexible audio.

I looked into Panasonic cameras briefly, but they're in the middle of updating their whole lineup, so their new models aren't available yet, and they don't have information on the old (discontinued) models online. After a great deal of searching, reading, comparing, and pondering, I decided on the Canon Vixia HF S20.

Features

  • According to most reports it works relatively well in low light.
  • It has a 3.5mm stereo mic jack.
  • It has Canon's Mini Advanced Shoe
  • 1080i (1920*1080) resolution, at 60 fields per second.
  • In "PF30" mode the camera produces 30p video, which is what I want for iMovie. Many other cameras cannot handle this, although iMovie 11 is less picky about what footage it can handle. PF24 (recorded at 24p) appears fine as well.
  • Still photography up to 3264*2456 (16:9, in video mode) or 3264*1840 (4:3, in still mode).
  • 32gb flash onboard.
  • 2 SDHC slots. I have a firmware file that should upgrade to SDXC (different filesystem with the same physical characteristics, for cards of 64gb and larger), but haven't tried it -- I don't have any SDXC media to test against. The camera does have a "Relay Recording" mode so video can overflow from onboard to slot A and then slot B, although I may never need this. I might not even use the onboard flash memory if my iMac's SD(HC) slot is much faster than the camera's USB port.
    • To import directly from SD cards (should be faster, and enables the camera to charge simultaneously), connect it to a flash reader, launch iMovie, choose Import from Camera..., and select the SD reader (my iMac's is called "Apple Internal Memory Card Reader" from the Camera: popup
  • 10x zoom -- less than some other cameras but still quite useful.
  • Popup flash/video light. Handy, although I'm not sure if this will ever be useful.

I used the S20 at a very dark show with the DM-100, and got usable (though decidedly grainy) video with clear sound.

Numbers

  • The included Canon BP-808 battery is rated at 55 minutes.
  • I bought a BP-819 with twice the capacity, rated at 105 minutes.
  • The S20 includes 32gb RAM, sufficient for 175 minutes of MXP (24mbps, 1920*1080) video.

Problems

  • The battery is frankly puny -- at highest quality, the built-in flash lasts more than 3 times as long as the puny battery. With the BP-819 as well, though, I should be able to record for 3 hours -- enough for any normal concert. For long events, I might need to bring the AC adaptor and plug the camera in. Unfortunately the camera can either run off AC current or charge the battery, but not both.
  • The S20 is overly complicated. It has 5 main modes:
    1. Dual Shot -- automatic mode, with both video and stills available, but no menu access at all.
    2. Video recording
    3. Photography
    4. Video playback
    5. Photo playback
  • Unfortunately Dual Shot mode does not allow overriding most of the defaults. This includes 60i frame rate (rather than the 30p or 24p I'd prefer), and won't even let me hide most of the onscreen status indicators.
  • Each menu command is linked to one or more non-auto modes. So to review all the settings, you must work through 3 menus in each of 4 modes.
  • The playback modes require choosing either photos or videos from one of the 3 possible sources (onboard flash, flash A, or flash B).
  • The menus are needlessly complicated. For example, when I'm shooting in "Dual Shot" mode, the steps to delete all photos from flash B are (this is different than the procedure to completely initialize flash B):
    1. Push the Camera/Review button.
    2. Push the Swap Playback(??) button (play arrow on 2 rectangles, surrounded by 2 curving arrows).
    3. Push B.
    4. Push Photos.
    5. Push Checkmark.
    6. Push Select All.
    7. Push Edit.
    8. Push Delete.
    9. Push Yes.
    10. (Wait for deletion.)
    11. Push OK.
    12. To shoot again, push Camera/Review again.
  • The touchscreen is not very sensitive. This was very annoying before I got used to it, but even now not all my presses register, and I worry about joggling the camera if I use the onscreen controls while filming.
  • No lens cap! Stupid! I cannot just throw a glass lens (or even a glass filter) in a bag and let it get scratched with every jostle. Fortunately it's 58mm, so finding caps isn't difficult, but I'm not about to pay $9 to Canon for a piece of plastic that should have been in the box anyway.
  • The dedicated low light mode is odd. It makes everything very blurry, as if it's using double-length exposures for pairs of frames. Not usable with any kind of motion. If I wanted to capture a still as video, it might serve, but the still camera would make more sense there.
  • The camera doesn't appear to work for streaming to a computer -- it only recognizes USB in 'playback' modes, not 'shooting' (recording) modes. This is a pity, as it means it can't work with iChat. I have a built-in iSight, but if I ever wanted to potcast, a better camera that isn't fixed into the monitor might be useful.
  • No viewfinder. This is a negative, but one I'm comfortable with. Canon's S21 adds a viewfinder and bumps the onboard RAM from 32gb to 64gb, but these are not worth an additional $235.

Conclusion

All things considered, I am happy with the camera. The touchscreen isn't very good and the menus are downright lousy, but I am able to mostly ignore them now that it's set up. I would have liked something simpler, but to get the odd combination of features I wanted, I needed a higher end and more complicated camera. Fortunately I can simply ignore most of the irrelevant capabilities (Direct burning of DVDs, onboard editing & effects, playlists, onboard creation of SD video from HD footage, etc.).

one comment

Friday, January 21 2011

Molly DNA at Pianos

By Chris Pepper on Friday, January 21 2011, 21:30 - music

Molly Mae and crew performed at Pianos, with a couple appearances by Latex Lily. Princess Sunshine opened, Amazing Amy contorted, and MDNA jammed with Squeezerock after their set. I got a bunch of decent photos.

  • Princess Sunshine opened with a rant on "Happily Ever after". Then she played accordion and ukelele and sang a couple songs about men.

Next, Molly Does Not Approve played a set:

  1. Cigarette Girl
  2. Free
  3. Find My Clitoris
  4. Canção Brasileira
  5. Pinup Girl
  6. Social Worker
  7. Baby

After Squeezerock's set the bands recombined.

  1. Hey Good Lookin' (extended jam with Julz-A -- unfortunately the lights went out partway through)
  2. Fortify (jam with Squeezerock; contortions by Amazing Amy)

Videos will be up on YouTube next week.

no comment

Sunday, December 26 2010

Brooklyn Blizzard!

By Chris Pepper on Sunday, December 26 2010, 22:00 - pictures

We got back as the much-anticipated blizzard was beginning. It snowed all afternoon and plenty stuck. Lots of shoveling!

blizzard!

one comment

Friday, December 10 2010

4th Street Nite Owls at Two Boots, December 2010

By Chris Pepper on Friday, December 10 2010, 11:59 - music

Blow, man, blow!

Tonight Gavin played clarinet in the 4th Street Nite Owls at Two Boots. The music was good and the photos & video (56:10) came out pretty well, considering the poor light -- although Gavin spent most of the show hiding in shadow.

I missed the beginning of the show, but saw most of the first set:

  1. Makin' Whoopee
  2. Sweet Sue
  3. Have a Little Dream on Me
  4. Louisiana Fairytale
  5. Jerry the Junker

and the whole second set:

  1. It's No Fun
  2. Stayin' Alive
  3. Nagasaki
  4. Crosspatch
  5. When Day Is Done
  6. Crazy about My Baby (and My Baby's Crazy about Me)
  7. I Wish I Were Twins

Video

The whole show (large & slow):

one comment

- page 1 of 17