Extra Pepperoni

To content | To menu | To search

Tag - synchronization

Entries feed - Comments feed

Thursday, December 11 2008

hb.sh (HandBrake Wrapper) Updated for v0.9.3

As threatened, I have an update for my HandBrake wrapper script. On a Mac or Linux system, the script examines one or more DVD folders (I rip mine with MacTheRipper), and converts the DVD MPEG2 content to MP4 files suitable for Apple TV (by default) or iPhone (with an iphone argument, e.g., hb.sh iphone). It's easy to hack up for different preferences, and will require changing some variables to match your preferred media folder layout. Thanks to Brian Beardmore for the original GPL script.

The required customizations are these two lines:

inputSearchDir="$HOME/tivo-inspector/input/$mySuffix"
outputDir="$HOME/tivo-inspector/output"

The main difference in v1.0.7 of the script is that it uses HandBrake 0.9.3's new improved presets, rather than hard-coding my custom settings. I've found the new presets give excellent quality in considerably less time.

If you have trouble compiling the Linux CLI, see my earlier post about v0.9.3 (I didn't try the GUI).

You can download hb.sh v1.0.7 (recommended), or copy & paste, but might have to fix line wrapping & remove blog cruft if you do.


#!/bin/bash

# hb.sh uses HandBrakeCLI to convert whole DVDs automatically.
# Copyright (C) 2007  Brian Beardmore
# Copyright (C) 2008  Chris Pepper

#  This program is free software; you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with this program; if not, write to the Free Software
#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

# This script requires HandBrakeCLI <http://handbrake.fr/>.

# This script searches the entire input directory directory tree looking
# for TS_VIDEO directories looking for DVDs to encode.  The output mp4
# filename is the DVD volume name.  When multiple tracks are to encoded 
# from a DVD, the output filename is appended with the track number.
# For example, 'Monsters, Inc.' DVD has both standard and wide screen tracks
# so there are 'Monsters, Inc.1.mp4' & 'Monsters, Inc.10.mp4' files created.
# For addition information on this script contact the script's original author,
# Brian Beardmore at medfk(at)realisticsoftware(dot)com or
# http://www.realisticsoftware.com
# Or contact Chris Pepper <http://www.extrapepperoni.com/category/television/>.

# Examples
# $ ./appletv.sh
#   [ encodes all DVDs mounted and encodes any tracks longer than 60min ]
#   [ to the mp4 folder in the users Movies folder ]
# $ appletv.sh -i ~/Movies -o toPod -b 1000 -w 576 --minGetTime 50
#   [ encodes all DVDs found in the users Movies folder and encodes any ]
#   [ tracks longer than 50min, with a 1000kb/s bitrate and scales the output ]
#   [ movie to a width of 576 pixels and puts the resulting mp4 files in the ]
#   [ toPod folder in the current directory that the command was executed from ]

# Syntax: HandBrakeCLI [options] -i <device> -o <file>
# 
# ### General Handbrake Options------------------------------------------------
# 
#     -h, --help              Print help
#     -u, --update            Check for updates and exit
#     -v, --verbose <#>       Be verbose (optional argument: logging level)
#     -C, --cpu               Set CPU count (default: autodetected)
#     -Z. --preset <string>   Use a built-in preset. Capitalization matters, and
#                             if the preset name has spaces, surround it with
#                             double quotation marks
#     -z, --preset-list       See a list of available built-in presets
# 
# ### Source Options-----------------------------------------------------------
# 
#     -i, --input <string>    Set input device
#     -t, --title <number>    Select a title to encode (0 to scan only,
#                             default: 1)
#     -L, --longest           Select the longest title
#     -c, --chapters <string> Select chapters (e.g. "1-3" for chapters
#                             1 to 3, or "3" for chapter 3 only,
#                             default: all chapters)
# 
# ### Destination Options------------------------------------------------------
# 
#     -o, --output <string>   Set output file name
#     -f, --format <string>   Set output format (avi/mp4/ogm/mkv, default:
#                             autodetected from file name)
#     -m, --markers           Add chapter markers (mp4 and mkv output formats only)
#     -4, --large-file        Use 64-bit mp4 files that can hold more than
#                             4 GB. Note: Breaks iPod, PS3 compatibility.
#     -O, --optimize          Optimize mp4 files for HTTP streaming
#     -I, --ipod-atom         Mark mp4 files so 5.5G iPods will accept them
# 
# ### Video Options------------------------------------------------------------
# 
#     -e, --encoder <string>  Set video library encoder (ffmpeg,xvid,
#                             x264,theora default: ffmpeg)
#     -x, --x264opts <string> Specify advanced x264 options in the
#                             same style as mencoder:
#                             option1=value1:option2=value2
#     -q, --quality <float>   Set video quality (0.0..1.0)
#     -Q, --cqp               Use with -q for CQP instead of CRF
#     -S, --size <MB>         Set target size
#     -b, --vb <kb/s>         Set video bitrate (default: 1000)
#     -2, --two-pass          Use two-pass mode
#     -T, --turbo             When using 2-pass use the turbo options
#                             on the first pass to improve speed
#                             (only works with x264, affects PSNR by about 0.05dB,
#                             and increases first pass speed two to four times)
#     -r, --rate              Set video framerate (5/10/12/15/23.976/24/25/29.97)
#                             Be aware that not specifying a framerate lets
#                             HandBrake preserve a source's time stamps,
#                             potentially creating variable framerate video
# 
# ### Audio Options-----------------------------------------------------------
# 
#     -a, --audio <string>    Select audio track(s), separated by commas
#                             More than one output track can be used for one
#                             input.
#                             ("none" for no audio, "1,2,3" for multiple
#                              tracks, default: first one)
#     -E, --aencoder <string> Audio encoder(s) (faac/lame/vorbis/ac3) 
#                             ac3 meaning passthrough
#                             Separated by commas for more than one audio track.
#                             (default: guessed)
#     -B, --ab <kb/s>         Set audio bitrate(s)  (default: 160)
#                             Separated by commas for more than one audio track.
#     -6, --mixdown <string>  Format(s) for surround sound downmixing
#                             Separated by commas for more than one audio track.
#                             (mono/stereo/dpl1/dpl2/6ch, default: dpl2)
#     -R, --arate             Set audio samplerate(s) (22.05/24/32/44.1/48 kHz)
#                             Separated by commas for more than one audio track.
#     -D, --drc <float>       Apply extra dynamic range compression to the audio,
#                             making soft sounds louder. Range is 1.0 to 4.0
#                             (too loud), with 1.5 - 2.5 being a useful range.
#                             Separated by commas for more than one audio track.
#     -A, --aname <string>    Audio track name(s),
#                             Separated by commas for more than one audio track.
# 
# ### Picture Settings---------------------------------------------------------
# 
#     -w, --width <number>    Set picture width
#     -l, --height <number>   Set picture height
#         --crop <T:B:L:R>    Set cropping values (default: autocrop)
#     -Y, --maxHeight <#>     Set maximum height
#     -X, --maxWidth <#>      Set maximum width
#     -p, --pixelratio        Store pixel aspect ratio in video stream
#     -P, --loosePixelratio   Store pixel aspect ratio with specified width
#           <MOD:PARX:PARY>   Takes as optional arguments what number you want
#                             the dimensions to divide cleanly by (default 16)
#                             and the pixel ratio to use (default autodetected)
#     -M  --color-matrix      Set the color space signaled by the output
#           <601 or 709>      (Bt.601 is mostly for SD content, Bt.709 for HD,
#                              default: set by resolution)
# 
# ### Filters---------------------------------------------------------
# 
#     -d, --deinterlace       Deinterlace video with yadif/mcdeint filter
#           <YM:FD:MM:QP>     (default 0:-1:-1:1)
#            or
#           <fast/slow/slower>
#     -5, --decomb            Selectively deinterlaces when it detects combing
#           <MO:ME:MT:ST:BT:BX:BY>     (default: 1:2:6:9:80:16:16)
#     -9, --detelecine        Detelecine (ivtc) video with pullup filter
#                             Note: this filter drops duplicate frames to
#                             restore the pre-telecine framerate, unless you
#                             specify a constant framerate (--rate 29.97)
#           <L:R:T:B:SB:MP>   (default 1:1:4:4:0:0)
#     -8, --denoise           Denoise video with hqdn3d filter
#           <SL:SC:TL:TC>     (default 4:3:6:4.5)
#            or
#           <weak/medium/strong>
#     -7, --deblock           Deblock video with pp7 filter
#           <QP:M>            (default 5:2)
#     -g, --grayscale         Grayscale encoding
# 
# ### Subtitle Options------------------------------------------------------------
# 
#     -s, --subtitle <number> Select subtitle (default: none)
#     -U, --subtitle-scan     Scan for subtitles in an extra 1st pass, and choose
#                             the one that's only used 10 percent of the time
#                             or less. This should locate subtitles for short
#                             foreign language segments. Best used in conjunction
#                             with --subtitle-forced.
#     -F, --subtitle-forced   Only display subtitles from the selected stream if
#                             the subtitle has the forced flag set. May be used in
#                             conjunction with --subtitle-scan to auto-select
#                             a stream if it contains forced subtitles.
#     -N, --native-language   Select subtitles with this language if it does not
#           <string>          match the Audio language. Provide the language's
#                             iso639-2 code (fre, eng, spa, dut, et cetera)

# Revision history:
# 1.0.7, 2008/12 -- Update for HandBrake 0.9.3, and use presets.
# 1.0.6 -- never released.
# 1.0.5, 2008/08 -- Require bash, per ebb.
# 1.0.4, 2008/06 -- Use different suffices so different flavors can coexist.
# 1.0.3, 2008/05 -- Add basic argument processing, so "appletv" and "iphone" as argument #1 produce different output.
# 1.0.2, 2008/05 -- hacks by Pepper to use HandBrakeCLI and optimize for AppleTV.
# 1.0.1, 2008/02/24 -- hacks by Pepper to use HandBrakeCLI and optimize for iPhone.
# 0.20070329.0 - initial release


#############################################################################
# globals

# const global variables
scriptName=`basename "$0"`
scriptVers="1.0.7"
scriptPID=$$
saveDVDinfo=1       # the DVD track info is also saved as .txt file when set
skipDuplicates=1    # if this option is off, overwrite existing files
E_BADARGS=65

if [[ $# = 0 ]]
 then
  myArgs='$HANDBRAKE_ARGS -Z AppleTV'
elif [[ $1 = "appletv" ]]
 then
  myArgs='$HANDBRAKE_ARGS -Z AppleTV'
  mySuffix=AppleTV
elif [[ $1 = "AppleTV" ]]
 then
  myArgs='$HANDBRAKE_ARGS -Z AppleTV'
  mySuffix=AppleTV
elif [[ $1 = "iphone" ]]
 then
  myArgs='$HANDBRAKE_ARGS -Z "iPhone & iPod Touch"'
  mySuffix=iPhone
elif [[ $1 = "iPhone" ]]
 then
  myArgs='$HANDBRAKE_ARGS -Z "iPhone & iPod Touch"'
  mySuffix=iPhone
else
 echo "Unknown format -- aborting!"
 exit 1
fi

# set the global variables to defaults
toolName="HandBrakeCLI"
toolPath="$HOME/bin/$toolName"
toolTrackArgs="-t 0"
toolArgs="-v"
inputSearchDir="$HOME/tivo-inspector/input/$mySuffix"
outputDir="$HOME/tivo-inspector/output"
minTrackTime="3"    # in minutes


# Pepper's sometime options:
# -s 1 # first subtitle
# -a 2 # second audio track ??
# For example (best within screen): HANDBRAKE_ARGS="-s1 -t11" iphone.sh


#############################################################################
# functions

parseProcessInArgs()
{
    if [ -z "$1" ]; then
        return
    fi

    toolArgs=""

    while [ ! -z "$1" ]
    do
        case "$1" in
            -h) displayUsageExit ;;
            --help) displayUsageExit ;;
            -i) inputSearchDir=$2
                shift ;;
            --input) inputSearchDir=$2
                shift ;;
            -o) outputDir=$2
                shift ;;
            --output) outputDir=$2
                shift ;;
            --minGetTime) minTrackTime=$2
                shift ;;
            *) toolArgs="$toolArgs $1" ;;
        esac

        shift
    done
}

verifyFindCLTool()
{
    # attempt to find the HandBrakeCLI if the script toolPath is not good
    if [ ! -x "$toolPath" ];
    then
        toolPathTMP=`PATH=.:/Applications:/:/usr/bin:/usr/local/bin:$HOME:$PATH which $toolName | sed '/^[^\/]/d' | sed 's/\S//g'`

        if [ ! -z $toolPathTMP ]; then 
            toolPath=$toolPathTMP
        fi
    fi  
}

displayUsageExit()
{
    echo "Usage: $scriptName [options]"
    echo ""
    echo "    -h, --help              Print help"
    echo "    -i, --input <string>    Set input directory to process all DVDs in it (default: /Volumes/)"
    echo "    -o, --output <string>   Set output directory for all output files (default: ~/Movies/mp4/)"
    echo "    --minGetTime <number>   Set the minimum time (mins) of the track/s to encode (default: 60)"

    if [ -x "$toolPath" ];
    then
        echo "   $toolName possible options"
        hBrakeHelp=`$toolPath --help 2>&1`
        hBrakeHelpPt=`printf "$hBrakeHelp" | egrep -v '( --input| --output| --help|Syntax: |^$)'`
        printf "$hBrakeHelpPt\n"
    else
        echo "    The options available to HandBrakeCLI except -o  and -i"
        if [ -e "$toolPath" ];
        then
            echo "    ERROR: $toolName command tool is not set up to execute"
            echo "    ERROR: attempting to use tool at $toolPath"
        else
            echo "    ERROR: $toolName command tool could not be found"
            echo "    ERROR: $toolName can be installed in ./ /usr/local/bin/ /usr/bin/ ~/ or /Applications/"
        fi
    fi

    echo ""

    exit $E_BADARGS
}

getTrackListLongerThan()
{
    # Two input arguments are are need. 
    #   arg1 is the time in minutes selector
    #   arg2 is the raw text stream from the track 0 call to HandBrake
    #   returns: a list of track numbers of tracks longer than the selector

    if [ $# -lt 2 ]; then
        return ""
    fi

    minTime="$1"
    shift
    allTrackText="$*"
    aReturn=""

    trackList=`eval "echo \"$allTrackText\" | egrep '(^\+ title |\+ duration\:)' | sed -e 's/^[^+]*+ //'g -e 's/title \([0-9]*\):/\1-/'g -e 's/duration: //'g"`

    trackNumber=""
    for aline in $trackList
    do
        trackLineFlag=`echo $aline | sed 's/[0-9]*-$/-/'`
        if [ $trackLineFlag = "-" ];
        then
            trackNumber=`echo $aline | sed 's/\([0-9]*\)-/\1/'`
        else
            set -- `echo $aline | sed -e 's/(^[:0-9])//g' -e 's/:/ /g'`
            if [ $3 -gt 29 ];
            then let trackTime=($1*60)+$2+1
            else let trackTime=($1*60)+$2
            fi

            if [ $trackTime -gt $minTime ];
            then aReturn="$aReturn $trackNumber"
            fi
        fi
    done

    echo "$aReturn"
}

makeFullPath()
{
    aReturn=""
    currentPath=`pwd`

    if [ $# -gt 0 ]; then
        inPath="$*"

        # put full path in front of path if needed
        aReturn=`echo "$inPath" | sed -e "s!~!$currentPath/!" -e "s!^./!$currentPath/!" -e "s!^\([^/]\)!$currentPath/\1!" -e "s!^../!$currentPath/../!"`

        # remove ../ from path - only goes 4 deep
        aReturn=`echo "$aReturn" | sed -e 's!/[^\.^/]*/\.\./!/!g' | sed -e 's!/[^\.^/]*/\.\./!/!g' | sed -e 's!/[^\.^/]*/\.\./!/!g' | sed -e 's!/[^\.^/]*/\.\./!/!g'`

        # cleanup by removing //
        aReturn=`echo "$aReturn" | sed -e 's!//!/!g'`
    fi

    echo "$aReturn"
}

isPIDRunning()
{
    aResult=0

    if [ $# -gt 0 ]; then
        txtResult="`ps ax | egrep \"^[ \t]*$1\" | sed -e 's/.*/1/'`"
        if [ -z "$txtResult" ];
        then aResult=0
        else aResult=1
        fi
    fi

    echo $aResult
}

#############################################################################
# MAIN SCRIPT

# initialization functions
verifyFindCLTool
parseProcessInArgs $*
# see if the output directory needs to be created
if [ ! -e $outputDir ]; then
    mkdir -p "$outputDir"
fi

# sanity checks
if [[ ! -x $toolPath || ! -d $inputSearchDir || ! -d $outputDir || -z "$toolArgs" ]]
 then

  if [[ ! -x $toolPath ]]
   then echo "ERROR: $toolPath is not executable!"
  fi
  if [[ ! -d $inputSearchDir ]]
   then echo "ERROR: $inputSearchDir is not a valid input directory!"
  fi
  if [[ ! -d $outputDir ]]
   then echo "ERROR: $outputDir is not a valid output directory!"
  fi
  if [[ ! -z "$toolArgs" ]]
   then echo "ERROR: $toolArgs is unset!"
  fi

    displayUsageExit
fi

# fix input and output paths to be full paths
inputSearchDir=`makeFullPath $inputSearchDir`
outputDir=`makeFullPath $outputDir`

# display the basic setup information
echo "$scriptName v$scriptVers"
echo "  Start: `date`"
echo "  Input directory: $inputSearchDir"
echo "  Output directory: $outputDir"
echo "  Minimum get track time: $minTrackTime mins"
echo "  Tool path: $toolPath"
echo "  Tool args: $toolArgs"
echo "  My args: $myArgs"
echo "  - - - - - - - - - - - - - - - -"

# find all the DVD videos in the input search directory tree
# spaces in file path temporarily become /008 and paths are separated with spaces
dvdTSVidList=`find $inputSearchDir -name VIDEO_TS -print0 | tr ' ' '\007' | tr '\000' ' '`

# process each DVD video found
for dvdTSDir in $dvdTSVidList
do
    # correct the tmp char back to spaces in the DVD file paths
    dvdTSDir=`echo $dvdTSDir | tr '\007' ' '`

    # get the DVD's name and path to root of the DVD
    dvdVolPath=`dirname "$dvdTSDir"`
    dvdName=`basename "$dvdVolPath"`
    dvdNameALNUM=`basename "$dvdVolPath" | sed 's/[^[:alnum:]^-^_]//g'`

    # display information
    echo "  * Processing DVD '$dvdName'"

    # create tmp link to the dvdVolPath to workaround a problem that
    # the HandBrakeCLI tool has a problem with spaces in the input
    # file paths in a script
    tmpNoSpacePath="/tmp/dvdVol-$dvdNameALNUM-$scriptPID"
    ln -s "$dvdVolPath" $tmpNoSpacePath

    # get the track list information from the DVD
    cmd="$toolPath -i $tmpNoSpacePath $toolTrackArgs /dev/null 2>&1"
    dvdTrackInfo=`eval $cmd`
    # save the DVD info
    outputFilePath="$outputDir/${dvdName}.txt"
    if [ $saveDVDinfo -eq 1 ]; then
        if [[ ! -e  $outputFilePath || skipDuplicates -eq 0 ]]; then
            echo "$dvdTrackInfo" | egrep '[ \t]*\+' > "$outputFilePath"
        fi
    fi
    # get the track number of tracks which are longer then the time desired
    trackFetchList=`getTrackListLongerThan $minTrackTime "$dvdTrackInfo"`
    if [ ! -z "$trackFetchList" ];
    then
        echo "   Will encode the following tracks: `echo $trackFetchList | sed 's/ /, /g'` "
    else
        echo "   No tracks on this DVD are longer than the minimum track time setting"
    fi

    trackCount=`echo $trackFetchList | wc -w`
    for aTrack in $trackFetchList
    do
        if [ $trackCount -gt 1 ]
            then outputFilePath="$outputDir/${dvdName}-${aTrack}.$mySuffix.m4v"
            else outputFilePath="$outputDir/${dvdName}.$mySuffix.m4v"
        # .m4v is important for AppleTV
        fi
        cmd="$toolPath $myArgs -i $tmpNoSpacePath $toolArgs -t $aTrack -o \"$outputFilePath\" > /tmp/${dvdNameALNUM}Results.txt 2>&1"

        if [[ ! -e  $outputFilePath || skipDuplicates -eq 0 ]];
        then
            # simple command execution
            #ripResult=`eval $cmd`

            # background command execution with some status

            echo "Command is: $cmd"

            eval $cmd &
            cmdPID=$!
            while [ `isPIDRunning $cmdPID` -eq 1 ]; do
                cmdStatusTxt="`tail -n 1 /tmp/${dvdNameALNUM}Results.txt | grep 'Encoding: '`"
                if [ ! -z "$cmdStatusTxt" ]; then
                    echo -n "$cmdStatusTxt"
                fi
                sleep 1s
            done
            echo ""
            wait $cmdPID

        else
            echo "   Output file SKIPPED because it ALREADY EXISTS"
        fi

        if [ -e /tmp/${dvdNameALNUM}Results.txt ]; then
            rm /tmp/${dvdNameALNUM}Results.txt
        fi
    done

    rm $tmpNoSpacePath
done

echo "  - - - - - - - - - - - - - - - -"
echo "  End: `date`"

exit 0

Sunday, November 9 2008

yum Killed Email, Again!

I ran "yum update" before bed last night to update my CentOS (RHEL) 5.2 system. Unfortunately, I didn't notice that email stopped flowing. Cyrus IMAPd was running, but no new mail was coming in. postfix was accepting mail, but new mail wasn't reaching Cyrus. Melissa let me know that mail wasn't working, and I discovered that again, the update had broken amavisd-new.

yum & rpm don't coexist well with cpan, and cpan itself has lots of trouble with Compress::Zlib. After a bunch of poking around, I eventually got cpan to reinstall (or just properly recognize?) Compress::Zlib, and got amavisd-new restarted; then postfix delivered the outstanding mail.

The whole cpan dance was prolonged and complicated by long cpan timeouts attempting to contact FTP mirrors -- presumably due to firewall restrictions that I never noticed because I avoid FTP. After a bunch of futzing with "o conf urrllist", I expunged the FTP mirrors and replaced them with HTTP mirrors; I was then eventually able to reinstall the necessary modules.

Thursday, November 6 2008

On the Job Market

I got caught up in Goldman's layoffs Wednesday. I hope to have a new job before the severance runs out, but if you are (or know someone who is) looking for a Linux/Solaris admin in the NYC area, please let me know.

My interests include writing & documentation, networking, Mac OS X, and open source.

My resume is online.

I actually wish this had happened a day earlier. It was a big comedown from the Obama victory the night before; in the reverse order, winning back the country would have been a nice counter to the layoff news.

Wednesday, August 20 2008

MySQL Initial Setup Crib Sheet (RHEL5)

Update 2008/08/22: There's actually a simpler command to create the database, once MySQL is secured and the account exists:

mysqladmin create newdatabase -u existinguser -p.


To test Movable Type, I needed a new MySQL installation on a CentOS 5.2 (equivalent to Red Hat Enterprise Linux 5.2) system. Here's a crib sheet with the steps I took to set up a new MySQL installation.

Get and Start the Software

  • yum install perl-DBD-MySQL mysql-server # Install MySQL server and the DBD perl module that Movable Type needs to talk to it.
  • service mysqld start # Start mysqld (the MySQL 'daemon', or server).
  • chkconfig mysqld on # Set mysqld to run at boot in future.

Secure MySQL

MySQL uses internal accounts which are totally separate from UNIX accounts. My MySQL installation came with 3 distinct root accounts (without passwords); a RHEL4 system configured MySQL with a pair of anonymous accounts! The MySQL RPM suggests securing the default accounts with mysqladmin, but the website points out that mysqladmin doesn't get all the accounts. Fortunately MySQL offers instructions on how to secure the initial accounts manually.

mysql> select host, user from mysql.user;
+----------------+------+
| host           | user |
+----------------+------+
| 127.0.0.1      | root | 
| localhost      | root | 
| mmm.reppep.com | root | 
+----------------+------+
3 rows in set (0.00 sec)

In WordPress, each blog has its own account and database (that's how I configure them, anyway). In Movable Type, a single account & database will be used for my whole Movable Type installation, which makes administration simpler.

  • Secure both root accounts by setting strong passwords.
  • Delete both anonymous accounts.
  • Create a new account for the blog.

To make sure I really did configure a required password for root, I logged out of MySQL and then tried to login without a password (which is how got in initially). This failed, telling me I had successfully disabled passwordless root access. Then I logged in as root with a password, to continue setting up MySQL tables. Note that I never supply passwords on the command line, because that's insecure. Instead I supply the password when prompted by the mysql command, which keeps it out of command history and ps output.

  • mysql -u root # Log into MySQL, which doesn't yet have a root password.
  • Delete the non-localhost root account.
  • Set passwords for root@127.0.0.1 & root@localhost.
  • Log out of mysql:
[root@mmm ~]# mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.0.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> drop user 'root'@'mmm.reppep.com';
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@127.0.0.1 = password('unencryptedpassword');
Query OK, 0 rows affected (0.00 sec)

mysql> set password for root@localhost = password('unencryptedpassword');
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye
[root@mmm ~]# mysql -u root
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
[root@mmm ~]# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.0.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> exit;
Bye
[root@mmm ~]# 

Create a MySQL Database & Account for Movable Type

mysql> create database movabletype;
Query OK, 1 row affected (0.01 sec)

mysql> grant all on movabletype.* to movabletype@127.0.0.1 identified by 'unencryptedpassword';
Query OK, 0 rows affected (0.00 sec)

Backups

I'm not covering MySQL backups here, but I use http://sourceforge.net/projects/automysqlbackup/.


Extras

  • use movabletype;
  • create user dotclear@localhost identified by '****';
  • grant all privileges on dotclear.* to dotclear@localhost;
  • show databases;

Monday, August 18 2008

Red Hat Kickstart without DHCP

Red Hat Kickstart is the right way to install RHEL, but Red Hat's Linux documentation assumes the host can boot via DHCP. If DHCP (or bootp) isn't available, there are some obscure options to provide the network configuration on the linux command line (at the boot: prompt). These options are ip, netmask, gateway, and dns, as in:

linux ks=http://10.10.10.100/kickstart.cfg ip=10.10.10.101 netmask=255.255.255.0 gateway=10.10.10.1 dns=10.10.10.100

I found several pages of kickstart docs which are missing these options, but they are listed on http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Installation_Guide-en-US/s1-kickstart2-startinginstall.html.

Thursday, August 7 2008

UltraEdit

I've been a serious BBEdit user for years (I suspect a beta password many years ago was a joke at my expense). I use vi daily but am not a fan, and I find emacs inexplicable. As an system administrator, editing text files (typically configuration files and scripts) is a large part of my job.

So spending my days using Windows, with no BBEdit, was a concern.

I used kate a bit, and it's not bad, but it's limited (cannot even compare 2 windows!), and Exceed's Copy & Paste support is extremely erratic.

Many people suggested UltraEdit, but due to an installer issue, I was unable to use it. That issue has been fixed, and I got it running today. I'm quite impressed, although I have already discovered that its sorting capability is downright feeble compared to BBEdit's GREP-enhanced sorting. I see that UltraEdit offers 4 different flavors of Find & Replace: Plain, perl regex, UNIX regex, and UltraEdit regex. This seems crazy to me -- I consider anything that's not 100% backward-compatible with PCRE a bug, but I am not selling to a population of users who live in MS Word.

Truly weak: Find for $ doesn't work properly in perl regex mode with UNIX line endings.

I am both impressed and mildly aggravated. UltraEdit is much better than anything else I've used on Windows (or Linux or Solaris), but it's also less polished than BBEdit. I'm not sure how much of my frustration is because I have the advantage of years of experience of BBEdit -- compared to only a few hours to learn UE so far -- and how much is real deficiencies and lack of polish in UE.

Monday, August 4 2008

Indirection in Configuration Management

"Give me a place to stand and a lever long enough and I will move the world."

I was grumbling under my breath at a configuration management system today, and reminded of this wonderful statement by Archimedes.

Configuration management is the discipline of building systems which manage other systems -- cfengine is a well-known open source example. I needed to reboot a few hosts on a regular schedule -- easily handled in 5 minutes with "vi /etc/crontab" on each, or an ssh loop to append to the crontab on each affected system. I was struck by how many levels of indirection I needed to traverse to get this done with configuration management. This in turn prompted some thought about why jumping through the various hoops was worthwhile.

There are many excellent reasons to use configuration management:

  • Time savings -- over repeating the same actions over and over; this increases with the number of hosts involved.
  • Consistency -- configuration management ensures that (portions of) systems which should be identical really are.
  • Reproducibility -- because CMS is naturally tied into version control, it is easy to either examine or recreate the state of affairs at an arbitrary time in the past.
  • Modeling -- a CMS ends encompasses a representation of all the systems it manages. This efficient representation of those systems is quite useful for examining and comparing them. It's especially useful with a large or dynamic population of administrators, as it provides a single place to learn about the whole constellation of systems, and enforces some consistency among the various ways admins can manage systems.

In the simplest case, to make a machine reboot once, I could pull the plug and put it back (assuming I was near, or could get to, the machine). In a non-CMS scenario, I would do it with ssh and the shutdown -r. In this case, it was considerably more involved:

  • Launch PuTTY.
  • Log into a system with a checkout of the CMS configuration files.
  • Find the appropriate file (non-trivial if the managed constellation is complicated).
  • Fetch the latest version of the file (with multiple users, it's unlikely my checkout is current).
  • Edit the file corresponding to /etc/crontab or /var/spool/cron/root (I used kate, as I don't enjoy either vi or emacs, and BBEdit wasn't available); kate popped back an X11 session tunneled through ssh.
  • Create a pair of local machine sets in the file (cfengine calls these 'aliases'), each including half the covered systems (the systems reboot at staggered times, so they're not all down at once).
  • Create the pair of crontab lines, one for each machine set, embedding the pair of different reboot times and the shutdown -r command.
  • Check the modified crontab file back into the version control system; enter a message for the change log.
  • In a distributed CMS, staging hosts pick up the changes from version control, either on a schedule or when manually kicked for emergency/rush changes.
  • The affected hosts pick up the change from the CMS, and implement the specified change.

The reason Archimedes' quote is apropos is that configuration management provides excellent leverage -- I can edit one file in one place, and easily affect several systems (potentially hundreds or thousands). Each hoop I have to jump through provides an additional fulcrum. I can sit at my desk and use PuTTY to log into dozens of systems, across the world -- without even knowing where they are. Each change I make to the version control system is automatically picked up by every host participating in the system, and available to every admin with a checkout. I don't have to log into 8 machines (even uninteractively) to make them reboot -- I can orchestrate it all from my local workstation.

Unfortunately, mistakes are leveraged too; there is often no good way to test changes to production systems during business hours. If the changes are restricted to non-production hours, when the admin might not be around to monitor them (and shouldn't have to -- it's an automated system, after all!), the window could be closed by the time the admin sees whether the change was successful. Missing a change window can easily defer a change 24 hours.

Sunday, July 13 2008

Today's Linux tip: "yum localinstall"

I needed to install the Citrix ICA client on CentOS 5.2 (RHEL 5.2), but it has very strange dependencies -- it complains about a version of libXaw which is present, demands an older version of libXm, and requires manual installation of openmotif 2.2.

The trick (thanks, FriedChips!) was yum --nogpgcheck localinstall ICAClient-10.6-1.i386.rpm, rather than rpm -Uvh yum ICAClient-10.6-1.i386.rpm. This way yum chased the dependencies for me, and didn't refuse to install the unsigned Citrix package.

Next I associated launch.jsp with /usr/lib/ICAClient/wfica.sh -- Citrix should have used .ica instead, because .jsp is used for other things. IIRC, EMC NetWorker used .jsp to launch their graphical console.

Unfortunately the ICA client insists on being wider than the physical display, but I can work around that. I wonder if it's because I simultaneously connected to the same XP system via RDP from both Linux and a Mac with different resolutions.

Update: Citrix is fixed on the size of my MBP's 1440*900 main display, which means it doesn't fit properly on the MBP's external 1280*1024 (or landscape 1024*1280) or my Linux box's 1280*1024.

Annoyingly, Citrix assigns the Mac's Command key to Alt on the Windows host. This doesn't work well, because although they avoid most Command key combinations in the ICA Client, Command-Tab switches Mac apps rather than Windows windows. Guys, just use the Option key! It even says alt on it, and nobody needs that key for Mac specific functions! Today's happy discovery: Command-Option-Tab switches Windows apps.

Next I have to figure out how to de-assign Alt-Tab from switching virtual workspaces in KDE. Copy & Paste don't work consistently when connected from KDE either, presumably because some events are being interecpted locally and others are being passed through. I won't need to use KDE as a Citrix terminal for much longer, though.

Crud. After all that, the Citrix ICA client doesn't display most text, making it useless. I can get some things to display by selecting them, but many things (including dialog boxes) are un-selectable. Junk!

Wednesday, July 9 2008

reppep service interruption

Ouch! At 10:31pm last night, I started patching both Linux servers running reppep and associated domains, prompted by Rich's BIND alert. At 12:33am, www.reppep.com finished installing approximately 255 CentOS patches (including BIND), and I rebooted. Everything looked fine, and I went to bed. This morning, I thought it a bit odd that I didn't have any new email, but not that unusual.

Melissa left me a message that mail wasn't flowing, but I couldn't fix it at work. Tonight I discovered that amavisd-new, which handles filtering for reppep email, was unable to start. Strangely, it was complaining about the Compress::Zlib perl module, which was actually installed (version 2.008, via the perl-Compress-Zlib-1.42-1.fc6 RPM). Some more digging indicated Scalar-List-Utils-1.19 needed to be reinstalled, which enabled amavisd-new to start (it checks for Compress::Zlib and refuses to start if it finds something wrong, which was apparently triggered by the Scalar-List-Utils issue).

mailq showed me postfix was now getting errors from amavisd-new about MIME::Parser and File::Temp. CPAN reinstalled MIME::Parser and said File::Temp was already current.

I bounced amavisd-new again, and tried postfix flush. Over the past 15 minutes, postfix has delivered the ~~650 outstanding messages, and all seems well.

Separately, Alex noticed our blogs were inaccessible, but bouncing BIND tonight cleared that -- odd, as I checked http://www.bertpepper.com/ and got valid DNS resolution from both nameservers immediately after patching, but obviously something I didn't notice was still scrambled.

Anyway, at 8:45pm, all seems present and correct.

Sorry for the disruption!

Thursday, May 8 2008

Batch Encoding with HandBrake

I use Handbrake to rip DVDs for my iPhone and Apple TV (and previously for my Treo 650, at much lower quality). It's excellent, but takes a long time, and slows down the whole system while compressing. My PowerBook G4 wasn't powerful enough to do H.264 compression in reasonable time, so I got into the habit of ripping the DVDs on the Mac with Mac The Ripper, copying them to the server, converting on the server, and copying back down to the iPhone through iTunes on the laptop. Transferring a full DVD via gigabit Ethernet takes under 10 minutes, which is why we have 3 1/2 gigabit Ethernet switches (8-port, a pair of 5-ports, and the 3 ports built into the Time Capsule) in a 3 1/2 person (2 grown-ups, a 5-year-old, and the cat) home.

Since then I've upgraded the laptop and replaced the server, so they're now both 2.4GHz Core Duos, but I still prefer to avoid the load on my personal machine -- especially now that we have an Apple TV, which likes large high-quality video files. The MacBook Pro runs HandBrake.app (Mac GUI), while the server runs HandBrakeCLI (Linux).

HandBrakeCLI is designed to convert individual files and is awkward for converting whole DVDs. Fortunately I found MediaForkCLI-batch.sh, a wrapper which is good at converting all the real titles from one or more DVDs; it automatically ignores short titles such as previews, and titles that have already been transcoded. Unfortunately MediaForkCLI-batch.sh is languishing, and hasn't been updated for HandBrake v0.92, but it was straightforward to update it to the new syntax.

In the process I made some changes and added an alternate version with different settings optimized for our Apple TV. On the Linux box (named inspector), the scripts are ~/bin/appletv.sh & ~/bin/iphone.sh. I put newly ripped files into ~/tivo-inspector/input/ on inspector, and these scripts rip whatever they find there, generating files in ~/tivo-inspector/output/.

On the MacBook Pro, I keep video files in ~/tivo, and ~/tivo/tivo-inspector is a symbolic link to tivo-inspector on inspector, the server -- mounted via AFP. This makes it easy to move files back and forth.

For convenience, I have a couple aliases on the MacBook Pro which log into the Linux machine, start a screen session, and run the appropriate script. This means that if I disconnect the laptop, I can later ssh back into the server and use screen -DR to reconnect to my running conversion session. When the conversion is done, the aliases open the remote folder in the Finder. I move the files back up out of input & output, drop them onto iTunes, and then either Sync the Apple TV (which gets copies of my whole iTunes Library), or sync them to the iPhone.

The scripts are appletv.sh & iphone.sh.

If you use these scripts, you should carefully review the encoding settings. In particular, I encode Apple TV audio to AAC stereo, because our Apple TV is connected to a pair of stereo speakers with a subwoofer; they cannot handle Dolby Pro Logic, which is better for most TVs.

You will also have to update paths in the scripts -- to the input and output directories, and to HandBrakeCLI itself -- to make them work on your own system.


Here are the aliases in sh/bash format; change the equal signs to spaces for csh/tcsh:

alias appletv="ssh -t inspectore time screen bin/appletv.sh; open ~/tivo/tivo-inspector"
alias  iphone="ssh -t inspectore time screen bin/iphone.sh;  open ~/tivo/tivo-inspector"

Tuesday, March 18 2008

kate Is Not Bad

I got Exceed, and after reassigning my left modifiers to X instead of Windows, kate is quite reasonable. It uses kompare for graphical diff, and comes with some CVS plugins. I prefer BBEdit's diff display (although BBEdit's diff has been broken for years). I'm not sure how I managed to view a couple windows in xemacs from kompare, but I can probably avoid that in the future...

kate is clunkier and less featureful, and not as configurable as I expect -- the commands I want to assign to the toolbar, for instance, are not available in that context. On the other hand, BBEdit doesn't use a toolbar at all, and rearranging menus is only supported in limited ways, so I'm not convinced kate is inferior here -- it may just feel like that to me as a BBEdit user. Hopefully Subversion support is available for kate, but that doesn't actually matter to me right now.

I need to get Copy & Paste working between Windows (including PuTTY) and Exceed; hopefully this will be straighforward, but it doesn't just work.

Per IDM, UltraEdit cannot be installed without admin rights, which I do not expect to get, so that's out -- at least until they offer an alternate installer.

kate icons are a bit fuzzy, but they fit the Linux aesthetic, and the fonts are very nice.

Wednesday, March 12 2008

Extra Pepperoni Re-Hosted

After DreamHost's breach 8 months ago, I was aggravated at their poor handling of the situation, but willing to give them the benefit of the doubt, and still happy with their low prices and flexible services.

With the new bad news and worse confirmation (still with poor incident handling), though, it's time to get out of dodge.

I have moved Extra Pepperoni back onto my own hardware. I started blogging on Apple's Blojsom install, but gave up on Tiger Server for Blojsom (and Mailman) because the services kept silently shutting down, leaving me to notice they were disabled days or weeks later (no fault of Blojsom or Mailman -- Apple didn't do a good job porting SpamAssassin either). Bringing up a WordPress blog and mailing lists at DreamHost was easy and cheap, but that's no good if they are unsafe.

I'll look at moving a couple very light-duty Mailman lists off DH next, but the lists are so lightly used I'm not too concerned. There just isn't any confidential information on the mailing lists, aside from their tiny subscriber lists.

Ah, well. I now know much more about WordPress and MySQL than I cared too, but the setup wasn't too bad. I hadn't realized how many customizations and tweaks I made to WordPress until it came time to recreate them on my own system:

  1. Almost Spring theme (included by DreamHost); with minor hack
  2. PHP Markdown Extra; with minor hack
  3. MySQL admin UI
  4. WP-DB-Backup (DH included one, which I'm no longer using)
  5. mod_rewrite for permalinks
  6. Admin-SSL, with "Shared SSL" tweak, integrated into my existing SSL site (meaning EP is available through two different "sites", and I have to keep the Apache configurations reconciled)
  7. Twitter
  8. WP-Cache (DH standard)
  9. Akismet anti-spam registration
  10. Technorati pinger (came over automatically with the DB).
  11. Fix for widget.php to use legal JavaScript tag.

Thursday, February 21 2008

Between Jobs

For dessert: 4 bags of chocolate chips

It feels very very strange to be unemployed -- it's been 7 years since the last time, and I was too freaked out at Shooting Gallery laying me off to feel this way. Now that I'm a grown-up (having kid(s) means you're responsible, even when you're irresponsible!) it's a good thing that we're covered by RU insurance past the start date for GS insurance, but the whole experience is still very odd. I wiped the third computer today at 5:30pm, and am copying data off computer #4 (old reppep.com) right now in preparation for retiring it (it's falling apart, apparently -- optical drive died an hour ago).

Now I just need Apple to update the MBP15s, so I can replace this PowerBook. It's doing better than I thought, though -- doesn't seem any doubt that it will serve until the next update.

RU IT did right by me today -- a grand spread, consisting of John's pizza, baby back ribs, beef ribs (they looked like something from The Flintstones), and chicken wings. A nice (short) speech by Armand, and well wishes all around. Elaine hung a bunch of colorful signs, which delighted Julia.

I closed out my helpdesk tickets, turned in my keys (forgot to turn in my ID/swipe card, though), and updated the documentation on our load balancers again, as well as re-re-recapping for my co-workers. I had to say "Look, when you feel like you're an idiot, don't worry -- I felt like that repeatedly for years while working with these. The Big-IPs are absurdly complicated. Two kernels, a super 'switch card' that's doing all kinds of crazy (non-switch) stuff, over 20 IP addresses, 8 networks, plenty of bugs, and delays in getting technical support. It's not you!"

Maybe I'll have some time to investigate Linux & Windows text editors.

Tuesday, February 19 2008

reppep.com Migrated

On Feb 19, 2008, I shut down the old reppep.com server, which ran Mac OS X 10.4 "Tiger" Server, and replaced it with a new (cheaper and faster) PC running Linux. Unfortunately, the password formats are incompatible, so I apologize to app reppep users for the disruption.

Please call me if you have an account on reppep.com and haven't received your password already, or find anything not working right.

I switched from Apple's jabberd to Openfire, which doesn't use the UNIX system accounts, so let me know if you want a chat account (compatible with iChat & GTalk).


[Done] I forgot SquirrelMail address books -- should be able to bring those over too.


  • Firewall problem fixed. SMTP MX issue fixed.
  • Virus filtering problem fixed.
  • Webmail certificate fixed.
  • Quota problem fixed.
  • Virtual domains for email fixed.

As of 5pm, I don't know anything that doesn't work (aside from SquirrelMail address books) [fixed Thursday].

Thanks for your patience!


As of 10:30 on the 20th, things seem to be working. Something's screwy with amavisd-new's quarantine, but mail is going through. I reinstalled Openfire, and chat seems okay under the correct hostname/certificate name now (will try signing it as ca.reppep.com later).

Good timing -- the optical drive on the old server died tonight.

I have distributed all the new temporary passwords, so any users having trouble logging in should let me know.

Markdown.cgi is still broken, but I'm the only person who uses it here, so I'll get to it.


On Thursday the 21st, I found a problem with amavisd-new -- it had quarantined 32,000 messages in a single directory, and was stuck (apparently ext3 doesn't support more than 32,000 files in a directory). I cleared it out and finally managed to disable quarantine, which wasn't as easy as it should have been, and the backlog of messages have been delivered as of 9:15pm.

At 11pm, I fixed an issue preventing SMTP AUTH from working properly, which was interfering with sending email to non-reppep addresses.

Monday, February 18 2008

System Admin Interview Questions

I was quite impressed by Joel's description of the hiring process, and we've been doing a lot of interviewing for System Admins lately. I put together a list of standard questions to ask during interviews, which has been quite helpful in judging a) how much technical knowledge people have, and b) (just as important) how good a match they are for the skills void we were trying to fill at the time. Here they are, for the next person who needs to perform a similar exercise.

  1. How many systems does your team manage (Linux, Solaris, Windows, etc.)?
  2. How large is your team?
  3. Which OS are you most comfortable/familiar with?
  4. Which Linux flavors are you most comfortable/familiar with?
  5. Which Red Hat versions are you familiar with?
  6. Are you familiar with kernel programming or configuration?
  7. Have you done any custom packaging or kickstarting?
  8. Have you used or managed Sun JumpStart?
  9. How much experience do you have with Sendmail?
  10. ... NetWorker? Version? Managing backups, or just configuring clients?
  11. ... LDAP? Brand & version? LDIF or just querying?
  12. ... firewalls (iptables, ipf, etc.)?
  13. ... network administration (Cisco, sniffing, etc.)?
  14. ... Apache httpd?
  15. ... Tomcat & Java?
  16. ... EMC (Clariion, PowerPath)?
  17. ... shell scripting, and with which shells?
  18. ... perl scripting?
  19. ... Veritas VM/FS? Versions?
  20. ... Veritas Cluster, or other HA? Versions?
  21. ... snapshots? In which products?
  22. ... load balancing
  23. ... Oracle (as SA, not DBA)?
  24. ... HPC?
  25. Please briefly explain the difference between RAID 1 and 5. What are layered RAID levels, and when are they appropriate?
  26. What sizable projects have you done recently?
  27. Why are you leaving your current employer / did you leave your last employer?
  28. Please give specific examples of some routine tasks you've performed recently.
  29. Have you done systems specification and design (servers, multi-server configurations)?
  30. Have you worked with customers directly, or primarily with/for other IT personnel?

It didn't make sense to publish a list of questions when I was involved in the interviewing process, but now that I'm leaving Rockefeller and no longer interviewing UNIX Admins for them, I can post my sample questions.

Saturday, January 5 2008

Cyrus IMAPd: only about as complex as a USENET news server

For several years, I've been saying Apple made a bad choice when they picked Cyrus IMAPd as the POP/IMAP server for Mac OS X Server. It's a huge and complicated system, encompassing IMAP, POP, SSL, Sieve filtering, LMTP delivery, USENET news, clustering/proxy (Murder), pluggable authentication (SASL), etc. I cannot think of a single company outside Cupertino where it would make sense to run an enterprise mail system on Mac OS X Server, but Apple continues to add these inexplicable high-end features to its mail server, most recently XSan-based email clustering in Leopard Server.

The statement that convinced me (shortly after I had migrated to Cyrus IMAPd on Mac OS X Server 10.4 "Tiger") that I would never choose to run Cyrus for my personal use, was the following -- which I came across again today:

Installation Overview

This system should be expected to have the same order-of-magnitude installation complexity as a netnews system. Maintenance should have similar complexity, except administrators will have to deal with creation and deletion of users and will have the option of managing quotas and access control lists.

USENET news is infamously demanding and bandwidth intensive. It would be wonderful if Apple had taken Cyrus IMAPd, repackaged it (without too many changes!), and put a powerful and simple interface on top. The did this quite successfully with Apache httpd (although Server Admin breaks down on complicated configurations and has obscure bugs). Lots of people use Mac OS X Server to run websites and think it's easy & simple. Considering the typical reactions of those same people to the httpd .conf files "under the hood", this is a noteworthy triumph. Similarly, Time Machine provides a reasonable approximation of scheduled snapshots on a high-end NAS for do-it-yourself file recovery, with a simple interface that insulates users from the nitty-gritty of copy-on-write and hard links.

Cyrus did not get as much attention, though. Basically, Apple makes it pretty easy to create email accounts, provides a Repair button for the overall Cyrus database, and provides a Reconstruct button for individual accounts. That's about it. Unfortunately, Apple doesn't really document maintenance beyond "press the button and it will fix your problem". I've had several serious database problems which Apple's Repair button did not help with. Those were bad times.

Similarly, I have had problems where users could not log in, but Workgroup Manager claimed their accounts were usable. I eventually discovered that resetting passwords with passwd works sometimes, and re-setting passwords in Workgroup Manager works consistently, but when I asked Apple about it, the eventual response was basically, "Yes, that's bad; you should restore your accounts from your recent Open Directory export." Not a good answer.

It doesn't help that Apple's SpamAssassin and ClamAV installations are broken, as these result in more spam and slower deliveries.


So why am I planning to migrate to Cyrus IMAPd on CentOS 5.1? Well, I'd really like to just copy my 5gb mail directory to the new system and have my clients not notice the difference. Eudora doesn't handle (IMAP) change well -- renaming a single IMAP directory can force it to download all messages again, and various other things can cause Eudora to lose date stamps on sent mail, or message state information (when it gets disassociated from the actual message on the IMAP server). If I can make Cyrus work, I'll be very happy, and if I can't I'll try Dovecot (Red Hat's default) or Courier (which I hear is also good).

Also, I know it can work, and I have a rough model to work from on my Tiger Server, but if I wasn't using Cyrus already I would stay away from it, as I wish Apple had done.

Thursday, December 20 2007

Installing Linux: NFS vs. HTTP

I'm digging deeper into CentOS (basically a free version of Red Hat Enterprise Linux) v5.1, and for me that entails dozens of runs through the installer, testing out kickstart configuration variations.

This led me to wonder if it is faster to install via NFS or HTTP. I couldn't find a useful answer online, so I ran a couple simple tests. My client is a 2.4GHz Dell PowerEdge 600SC using SATA disks on a Promise TX4. My server is a dual 1.25GHz Power Mac G4, running Mac OS X Server 10.4.11. They're connected via private network, using a NetGear gigabit Ethernet switch.

For NFS installations, anaconda takes a directory containing a DVD ISO (or set of CD ISOs), and automatically loopback mounts them as part of the installation process. This is very handy with the CD ISOs, as it doesn't require much configuration on the server -- just an NFS export.

In contrast, HTTP installation doesn't work against ISOs -- the web server must serve up the individual files, whether from a loopback mount on the server, or a directory where the files have been extracted. With the 6 CD ISO files, this is quite a nuisance; with the DVD ISO, it's not so bad.

My fairly complete kickstart configuration installs 2,103mb of packages. Installation times were quite similar, but a bit faster for HTTP, at 18:28 for package installation and 24:37 total. Via NFS packages took 19:57; total was 27:38.


For reference, I used the following partitioning configuration, which factors into total installation time:

part /boot --onpart=sda1 --fstype=ext2
part /     --onpart=sda2 --fstype=ext3
part swap  --onpart=sda3 --size=2048
part /var5  --onpart=sda5 --fstype=ext3
part /home6 --onpart=sda6 --noformat
part /sdb1  --onpart=sdb1 --noformat

The partitions already existed:

[root@pe ~]# df -hl|grep -v tmp
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             7.7G  2.8G  4.5G  39% /
/dev/sda6             664G  198M  629G   1% /home6
/dev/sda5             3.9G   73M  3.7G   2% /var5
/dev/sdb1             688G  198M  653G   1% /sdb1
/dev/sda1             251M  7.4M  231M   4% /boot

I used the following set of packages for testing:

%packages
@core
@base
@editors
@system-tools
@text-internet
@dns-server
@base-x
@network-server
@games
@web-server
@mail-server
@server-cfg
@admin-tools
@development-tools
@graphical-internet
@office
@development-libs
@x-software-development
@gnome-desktop
@graphics
@kde-desktop
@sound-and-video
kdegraphics
kdemultimedia
-bluez-gnome
-bluez-hcidump
-bluez-utils
-evolution
-evolution-connector
-evolution-webcal
-ibmasm
-krb5-auth-dialog
-mod_perl
-mod_python
-numactl
-openldap-clients
-openoffice.org-calc
-openoffice.org-draw
-openoffice.org-graphicfilter
-openoffice.org-impress
-openoffice.org-math
-openoffice.org-writer
-openoffice.org-xsltfilter
-pam_krb5
-pcmciautils
-php-ldap
-planner
-rp-pppoe
-sabayon
-sendmail
-sendmail
-sendmail-cf
-squid
-stunnel
-sysreport
-system-config-samba
-tux
-vconfig
-ypbind

Tuesday, December 18 2007

Leopard's bash auto-completion vs. symlinked directories

In Leopard, Tab completion in bash doesn't immediately append trailing slashes to symbolic links that point to directories. When I complained about this change to Apple, I was told it was user configurable, and I should just configure the old behavior. It took me a while to actually find the solution -- partially because it isn't within bash itself, and partially because it was quite a nuisance but not a serious problem.

For example, ~/www is a symlink to /Volumes/www, and I cd to directories below it quite frequently. I'm in the habit of typing "cd w[Tab]/pu[Tab]", which should expand to "cd www/public_html". This broke in Leopard -- I needed an extra Tab to get the / -- otherwise I'd end up with the ugly and non-functional "cd wwwpu". Anyway, the fix is:

echo "set mark-symlinked-directories on" >> ~/.inputrc

Now Tab completion works the way I want it to again. Thanks, stylishpants!

Saturday, December 15 2007

RHEL 5.1's "linux rescue" mode doesn't include full logical volume support

Red Hat has, with good consideration and foresight, been pushing people to use logical volume management for a while. It's not completely integrated into the RHEL5 installer, but they're pushing hard to make it ubiquitous, and telling people this is the right way to do things. Unfortunately, the syntax for specifying logical volumes within DOS-style partitions is still a bit obscure, and the manual page examples don't show the LV syntax; this is fixable, but will take time.

I used software RAID and LVM on my new installation, but it doesn't boot -- I've found several articles on making GRUB work with software RAID, so I believe I'll be able to get it working. The docs say I should be able to just use "lvm" (which is present) to get an lvm shell, but neither lvm nor lvm.static does anything -- they just dump me back in bash.

Fortunately, "linux rescue" finds my partitions (this time), but not being able to even list out physical volumes is worrisome.

I want mirrored /boot, but it's RHEL's mirrored /boot capabilities are pretty limited:

If you are making a RAID partition of /boot/, you must choose RAID level 1, and it must use one of the first two drives (IDE first, SCSI second). If you are not creating a seperate RAID partition of /boot/, and you are making a RAID partition for the root file system (/), it must be RAID level 1 and must use one of the first two drives (IDE first, SCSI second).

Speaking of LVM being immature, check out What is the process to fsck lvm volumes? in the Red Hat Knowledgebase:

First, boot into rescue mode by using the correct media. This is very important: When prompted to mounted the drives, do not. Using fsck on a mounted filesystem will destroy all the data on that file system. This is unrecoverable. The data will be gone forever--save for very expensive hardware-level data recovery.

Saturday, December 8 2007

Upgrading from Tiger Server to Linux

For over a year now, I've been following the development of Mac OS X Server 10.5 Leopard and testing betas, and anticipating upgrading reppep.com from Tiger Server on a dual 1.25GHz Power Mac G4 to Leopard Server on a dual 2GHz Power Mac G5. Over the weekend I had a change of plans, though.

Although I support Mac OS X Server at Rockefeller, I don't recommend it for most requirements, as Linux compares favorably for transparency (some of the MOSXS internals are unique and poorly documented), server software compatibility (although Macs are quite good here too), and price/features at the low end. A Core Duo Mac mini has plenty of juice to saturate our 768kbps/3mbps DSL circuit, but adding a couple drives more than doubles its price, and Apple's software RAID is quite broken; Linux software RAID is apparently quite good; I might eventually switch to hardware RAID. An Xserve is a great piece of hardware, but it's a bit exotic and I can get a fast generic PC cheaper; I don't want all the high-end features for a box that sits in our apartment.

Additionally, I've read perhaps 600 pages of docs on Leopard Server, and had at another 400-1500 yet to go. This is an investment I was finding hard to justify. The migration process is quite complicated, and Apple doesn't support migrating accounts from a Tiger system to a Leopard system -- I don't want to do an upgrade. I could clone the G4 to the G5 and upgrade it there, but I prefer to handle upgrades as scratch installations with manual migration of applications, so I know exactly what's been done. A lot of this is masked by upgrade procedures.

As part of this, I've decided to invest a bit more time in learning RHEL5 -- we have a couple systems at Rockefeller, but not much in production yet, and now seems like a good time to dig in some more.

Fortunately, all the services I've been using on reppep.com are available on Linux (and FreeBSD), so aside from another incredibly inconvenient password change cycle (for which it is arguably time anyway), the switch should be largely transparent to reppep.com users, although I still have plenty of research to do.

A brief timeline of reppep.com

  1. 1999: I left the National Audubon Society, and bought the Power Mac 7300 with accelerator card I'd been using there. I set it up with LinuxPPC and Apache, and started offering free web hosting to friends & family. LinuxPPC was eventually discontinued.
  2. I upgraded from LinuxPPC to Yellow Dog Linux, which was better than LinuxPPC, but had serious flaws.
  3. 2001: I was working on a couple remote FreeBSD machines (as admin of the Info-Mac server, and a user on the Apache Software Foundation userhost), and decided to learn more; I bought a cheap Celeron PC and installed FreeBSD 4.3 (IIRC); I upgraded through about v5.1 and a Pentium 4 (giving the Celeron box to the Info-Mac Archive, where it became the Info-Mac server for a while). I learned a lot about FreeBSD and UNIX in general, but eventually realized I was investing more time learning FreeBSD than I could justify. The best thing about FreeBSD is not a technical feature, but rather that the user community is so rich with knowledge. Reading the FreeBSD-STABLE list was amazing, as there was so much depth, freely shared with the community. While running on FreeBSD, I added mail services to the web services I had been offering. Note: Disruptions to personal email service are much worse than problems with personal web service.
  4. 2005: It became clear that I needed anti-spam, so I began researching SpamAssassin. While I was figuring out how to build the SMTP sandwich, with a public untrusted Postfix listener on port 25 & 587, and a filter, and then a listener on a high port like 10025 to accept and deliver mail to actual users, I installed a beta of Mac OS X Server 10.4 "Tiger", which had the whole thing implemented, plus ClamAV as a bonus. I started testing heavily before the release, and switched to MOSXS 10.4 shortly after it was finalized. It's been very good, but as time has passed, I've had more and more problems. In particular, Apple chose to use Cyrus as an IMAP/POP server, and Cyrus is complicated, but Apple ignores the complexity; this can make troubleshooting impossible. The SpamAssassin installation is slightly broken; it's a bit too old to offer the newer SpamAssassin self-upgrade mechanism. Server Admin is great, but has a bunch of bugs around SSL certificates, some of which destroy the certificates. Blojsom was nice, but Apple's installation was very unstable; I eventually moved my blog to WordPress hosted externally.
  5. 2008: I intend to switch to CentOS 5.1, which is basically a (legal) no-charge clone of Red Hat Enterprise Linux 5.1. This should make future upgrades a bit more straightforward, as I won't have to deal with Apple's Open Directory (OpenLDAP); it will also give me a bit more experience with RHEL5, which is a better investment for my time than Leopard Server.

- page 1 of 2