Users of our Isilon clusters need basic status information, so every 10 minutes our clusters run status.sh per /etc/mcp/templates/crontab. This provides a variety of useful information to users with access to the Isilon shared filesystem, and no need to provide shell access to the cluster nodes or remember the command syntax.

We now need to run some large/slow jobs, so I wanted a list of nodes in least-busy order. Obviously Isilon tracks this so SmartConnect can send connections to the least loaded node when using the "CPU Usage" connection policy, but it's not available to user scripts. The pipeline to provide a list of nodes sorted by lowest utilization to highest is applicable to all clusters, though -- just swap in the appropriate local cluster-wide execution command for isi_for_array.

status.sh

#!/bin/sh
# Record basic cluster health information

PREFIX=/ifs/x/common/cluster/status

isi status                   > $PREFIX/status.log
isi status -q -d             > $PREFIX/pool.log
isi job status -v            > $PREFIX/job.log
isi quota list               > $PREFIX/quota.log
isi quota list|grep -v :|grep -v default- > $PREFIX/quota-short.log
isi snapshot list -l         > $PREFIX/snapshot.log
isi snapshot usage | tail -1 > $PREFIX/snapshot-total.log
isi sync policy report | tail> $PREFIX/synciq.log
isi_for_array -s uptime      > $PREFIX/uptime.log
isi_for_array uptime | tr -d :, | awk '{print $12, $1}' | sort -n | awk '{print $2}' > $PREFIX/ordered-nodes.txt