Extra Pepperoni

To content | To menu | To search

synchronization

Entries feed - Comments feed

Saturday, November 28 2009

The Great Calendar Cleanup of '09

Amy and I have been sharing calendars through Google Calendar for a few years. We both have BusySync, and we have 4 family calendars, as well as one for our co-op (which I manage -- mostly just trash & recycling responsibilities), and some public calendar subscriptions.

This had gotten increasingly complicated over time, and Amy was aggravated that her iPhone & Google were often out of date because BusySync cannot sync when her MacBook is asleep (most of the time).

I came to grips with my anger at .Mac sync, which ate my data several times. Some of my problems were Palm-specific, and MobileMe has evolved sync somewhat since I gave up in 2007. I got a .Mac Family Pack (discounted), and set up our Macs & iPhones. Then I realized that iPhone OS 3 and iCal 4 (in Snow Leopard 10.6) both do direct Google Calendar sync, and kicked myself a bit. I like being able to update Address Book at work, rather than overwriting my Address Book data via rsync every morning (making it lose any local changes), so I will probably keep MobileMe for that, at least until the year runs out. I don't see myself paying for the ability to update Address Book (rather than Google Contacts) at work and Find My iPhone, although FMiP is cool.

Unfortunately, iCal's Google sync doesn't push alerts from iCal to Google, which is a deal-breaker for me, so I'm sticking with BusySync. BusyCal looks good, but I don't spend enough time on calendaring that it's worth maintaining or paying for another application. Amy doesn't use alarms, but she is still running Leopard, so she'll stick with BusySync at least until she upgrades. If I didn't already own BusySync for Amy, I might try subscribing to individual calendars from Leopard.

Side note: Google Calendar offers keyboard shortcuts.

Thursday, May 14 2009

CrashPlan: Deep Magic

Update 2009/05/15: The second part is not fully automatic. Sam did point his CrashPlan.app at my archives, but I didn't have to do anything on the client side. Still freaky!


Today I swapped backup drives with Sam. I have a Time Capsule (which hasn't failed much recently, to my surprise) for 'local' backups, and CrashPlan is for offsite backup. Sam discovered that CrashPlan is able to back up to a host behind a 'real' firewall. I'm not surprised that CrashPlan can use NAT-PMP to accept inbound backups, but I am surprised it apparently works through non-personal firewalls without manual configuration.

We believe Code42 must be running a heavy-duty proxying service, where each CrashPlan client connects to their servers, and their servers accept all the peer-to-peer backup traffic and dispatch it back down that 'inbound' connection. This is part of what makes IM work, but the traffic volume per user for CrashPlan is much higher than for something like AIM or Back to My Mac (screen sharing or individual file transfers). I hope this doesn't mean the software breaks if their server farm shuts down or is retired, and that they don't decide the intermediary service is too expensive (neither Sam nor I is paying for the service -- just for the CrashPlan+ licenses). If they really are retransmitting all p2p backups, code42 is passing double the non-local traffic of all their p2p users put together. They'd have to accept each byte from the client, then send it to the server. Backups across the local network don't need this. Perhaps CrashPlan is smart enough to only do this proxying trick for backup 'servers' behind uncooperative firewalls...

But tonight we got an even bigger surprise. When Sam plugged my 1tb drive into his Mac, both my Linux server and MBP immediately started backing up to the drive through Sam's computer. Note that I had not configured my systems to use his Mac. Apparently when he plugged the drive in, his Mac automatically registered the CrashPlan-created volume 'serial number' with the CrashPlan servers, and they automatically connected my clients to that drive (or perhaps it's all done via client serial numbers rather than volume numbers -- I'm just speculating here). Suddenly I had a new friend, 'Sanford' show up in both my CrashPlan clients. Freaky-deaky!

Sunday, January 18 2009

CrashPlan

For a while I kept off-site backups at Rockefeller, but I was never great about maintaining them.

CrashPlan attempts to make this simple and secure. Recently Sam asked me about CrashPlan, and we decided to host each other's backups. We each bought 1tb external drives, installed CrashPlan, and started kicking the tires. I noticed a few things.

The $60 CrashPlan+ license is per computer. Since CrashPlan automatically ties together all computers configured to share a personal account and the documentation talks about backing up between your computers, I thought the license was per account. $120 for a MBP and a Linux server is reasonable, but I was annoyed when I tried to install my license on the second computer and got a $60 error message.

The CrashPlanEngine grants control to anyone who can connect to it. This is bad, especially since my Linux server has a bunch of users on it.

Is it a bad idea to run the CrashPlanEngine as root? There are basic risks associated with it. These are the same as any service running as root. Because the engine is written in java, it is immune to buffer overflows, a common exploit for poorly written C code. You should be aware that any desktop client connecting has "permission" to select any file and back it up. Therefore, you should "require" authentication each time the UI is run. This is in the "settings/security" part of CrashPlan UI.

The installer failed to make /usr/local/crashplan/bin/CrashPlanEngine executable (fix: chmod +x /usr/local/crashplan/bin/CrashPlanEngine).

The installer only configured the CrashPlanEngine init script to run at runlevel 3. I might run at runlevel 5, and crashplan should still run, so I used chkconfig crashplan on.


Overall, I like CrashPlan so far. The UI is clean and includes everything I have needed. The design seems thoroughly reasonable. The encryption (stronger with CrashPlan+) gives me peace of mind. Seeding the initial backup over FireWire was fine -- we'll have to see real-world performance up our 768kbps DSL uplink.

Because we each have a significant amount of data, and moderate Internet connections at home, we are seeding locally -- performing the initial backups via FireWire & USB before exchanging drives. Afterwards, we'll update our own backups over the Internet -- CrashPlan supports this, and uses encryption to keep our data private, even from the partner who physically possesses the drive.

Our seeding is a bit complicated. Macs (my laptop & Sam's desktop) use HFS+. Linux (my server) uses ext3. FreeBSD (Sam's server) uses UFS2. This makes deciding what filesystems to put on the 1tb drives non-trivial, as they need to match the other person's CrashPlan server -- I will host Sam's drive on my Linux server, so want ext3; I don't know what format Sam wants, but hope it's not UFS2, as I no longer have a working FreeBSD server and Code42 doesn't support FreeBSD. CrashPlan has a doc on pre-seeding remote backups, but it's not terribly clear and assumes only 2 computers and a single filesystem format.

Wednesday, July 11 2007

.Mac synchronization, the free way (rsync)

Hooray! I now have a functional script for copying my iCal, Address Book, and Safari bookmarks. I suspect it's copying more than necessary, so I may refine it, but it's doing what I need now. I am amazed at how many places Apple sticks calendar data!

A big thank you to the rsync and OpenSSH developers. Once again, the script is largely comments, but that's good -- it is fundamentally simple.

Here is version 1.0 -- please check http://www.extrapepperoni.com/category/computers/synchronization/ and http://www.reppep.com/~pepper/code/ to see if there's a newer version.

I wrapped the lines here to make it more readable.


#!/bin/sh
# rsync-dotmac.sh
# v1.0, 2007/07/11 by Chris Pepper
# Check for a newer version at:
# <http://www.extrapepperoni.com/category/computers/synchronization/>
# <http://www.reppep.com/~pepper/code/>

for h in mac1 mac2
 do
  ssh $h rm -Rf ~/Library/Caches/com.apple.iCal ~/Library/Caches/iCal \
   ~/Library/Caches/Metadata/iCal \
   ~/Library/Preferences/com.apple.iCal.alarmsCache.plist \
   ~/Library/Preferences/com.apple.iCal.AlarmScheduler.plist \
   ~/Library/Preferences/com.apple.iCal.plist \
   ~/Library/Preferences/com.apple.iCal.sources.plist \
   ~/Library/Preferences/com.apple.iCalSync.AlarmScheduler.plist
  rsync -va --delete ~/Library/Application\ Support/AddressBook/ \
   $h:"Library/Application\ Support/AddressBook/"
  rsync -va --delete ~/Library/Application\ Support/iCal/ \
   $h:"Library/Application\ Support/iCal/"
  rsync -va --delete ~/Library/Calendars/ $h:Library/Calendars/
  rsync -va --delete ~/Library/Safari/Bookmarks.plist \
   $h:Library/Safari/Bookmarks.plist
 done

# To use, enter your read-only hostnames on the 'for' line above.
# They must, of course, be accessible from the system you run this script on.
# The script pushes data (Address Book, iCal, and Safari bookmarks) from
# a master to one or more read-only systems. It is intended as a free
# replacement for a part of Apple's .Mac service that caused me some
# frustration.
# I wish Apple provided a way to tell those applications not to allow
# (futile) changes on read-only systems, although with .Mac services
# that should not be necessary.
# I don't sync ~/Library/Keychains/, because my keychain items vary
# between systems, so unidirectional sync would be suboptimal.
# You may get warnings when going between OS versions; I have had no
# trouble just ignoring them (so far).
# If you get double birthdays, try disabling and re-enabling the
# Birthdays calendar in iCal's Preferences.

Monday, July 2 2007

How does .Mac notify applications that their data has changed?

Now that I've given up on .Mac sync, I see it's very simple to push the data out. The biggest problem is that rsync doesn't approve of Apple's spaces in names, so I have to get a bit silly with double quoting. Not a big deal.

This raises a big question, though: How does .Mac sync notify apps that their data has just been changed out from under them, and they should reload? Obviously all these apps have some mechanism to reread their data, because Apple uses it, but I have no idea what it is. For Address Book & iCal, it would be reasonable to close them as part of the script (if I could find a way that didn't involve sending a Quit AppleEvent, which has an annoying tendency to launch non-running apps solely so it can tell them to Quit). But for Safari, it would be quite bad to lose all the tabs I left open.

Does anyone know how .Mac does its update magic?

Here's the simple script -- insert your own names for mac1, mac2, etc.:

#!/bin/sh
# rsync-dotmac.sh
# Push or pull current .Mac content (Address Book, iCal, and Safar bookmarks) from master to read-only systems.
# Really needs a way to tell applications on read-only systems not to allow (futile) changes.

for h in mac1 mac2
 do
  rsync -n -va --delete ~/Library/Application\ Support/AddressBook/ $h:"Library/Application\ Support/AddressBook/"
  rsync -n -va --delete ~/Library/Application\ Support/iCal/ $h:"Library/Application\ Support/iCal/"
  rsync -n -va --delete ~/Library/Safari/Bookmarks.plist $h:Library/Safari/Bookmarks.plist
 done

Note that this script doesn't actually update anything. Once you have your real hostnames entered, remove -n from the script, and perhaps the v option as well, since you won't be seeing the output once it's in your crontab.


Hmm, I wonder if Library/Application\ Support/AddressBook/.database.lockN is relevant...

Wednesday, June 27 2007

Hating iSync

I give up on iSync. Last week it tanked again, and I cleared out all the Mac bits I could find and restarted from just my PowerBook (as well as I could), but it kept bloating and not finishing. On the third attempt, Conduit Manager started crashing during iSync.

Eventually I wiped my Treo, which is not an acceptable fix for Apple's data corruption. Now I'm trying to get everything back in sync.

I thought it was going to work, after a half-hour iSync run, but CM crashed again. What a load.

I think I just have to accept that Apple hasn't solved this problem (despite their claims), and switch to rsync updating my calendar, Address Book, and bookmarks from my PowerBook with rsync. So much for the (fantasy) of being able to update my calendar on home & work Macs.

Crud. Crud. Crud.


Conduit Manager keeps crashing on iSync. Now I'm wiping iCal and reverting. Yuck.


I had to wipe the Palm, but after Reverting iCal to a current backup, Conduit Manager miraculously stopped crashing (after 4 crashes, so it was a surprise).

Apple intends to renew my .Mac membership in 47 days, so that gives me a timeframe to wean myself from it. Murphy says I will do so, and then the iPhone or Leopard will have a great .Mac feature that sucks me back in...

Tuesday, June 26 2007

iSync Puking AGAIN!

Last week my PowerBook ran out of free space. After cleaning up and rebooting a couple times, I realized it was because ~/Library/Application\ Support/SyncServices/ was bloating again. I cleared it out, re-enabled iSync (deleting this folder disabled the iSync Palm HotSync conduit), and ran out of space. I deleted again & left iSync off, because I didn't have time to fight with it.

Tonight, I again cleared my .Mac sync data (unregistered all clients) and iSync history, rebooted, did a .Mac sync, and re-enabled Palm synching. Perhaps half an hour later, the Treo 650 just finished synching "for the first time" and SyncServices is 1.4gb. That was a "Fast sync"! No good.

pepper@pepperbook:~$ du -hs Library/Application\ Support/SyncServices/
1.4G    Library/Application Support/SyncServices/

Loading “Apple”
    Conduit “iSync Conduit” version 3.0.0
    Sync type is Fast
iSync Conduit starting 2007/06/26 12:30:55 AM
OK iSync Conduit

.Mac sync is still spinning in the menu bar, and SyncServices is now up to 1.5gb.

Saturday, May 5 2007

iSync Finally Behaving again

I have a lot of trouble with iSync (to my Treo) and .Mac sync. A few weeks ago, it was generating bogus complaints and bloating ~/Library/Application Support/SyncServices (up to 9gb in a single HotSync, which ran the PowerBook's hard disk out of space). I eventually cleared that one.

I had to abort a HotSync a couple days ago. Yesterday, Palm syncs (which I do twice per day or more, to get fresh plucker content) stopped updating my calendar. I cleared iSync's state, cleared .Mac's state, completely erased my Treo (scary, when the master data on the PowerBook might have been [and in the end actually was] corrupt), yada yada yada. I got different (related) errors, but nothing actually did the trick, until I backed up my data from iCal and reloaded it (luckily, this apparently blew away the corruption). Even this was fraught. I tested by loading it on another system -- trying to make sure the export was good, as I didn't trust the source. But when I attempted to open the iCal backup on the other system, iCal just spun and spun but never seemed to get anywhere. Eventually I realized it was calculating the disk size on the iCal backup. iCal backups are actually packages (folders), and it was taking much longer than it should have to total up the size of the folder, during which iCal was unresponsive (I actually force quit it a few times before I figured out what was going on).

Lesson for Apple is (aside from please fix synching in Leopard!!!): when complaining about a database synchronization error, identify the database you're complaining about! The iSync conduit was spewing errors about updating a specific record number. Of course, I have no reference for what that record number means. Had it mentioned that this was a problem in the iCal database, not the Palm DB or the "Truth database" (on iDisk), I could have saved hours and skipped erasing the Treo.

Another bug, just confirmed and reported. In the .Mac sync reconciliation dialog, I can't activate the "default" (blue throbbing) button with Return/Enter; it gets passed through to the window behind the dialog, although I can change the dialog's pop-up menu from the keyboard.


As it turned out, i started this posting prematurely last month. iSync was still corrupted. I had to wipe my Palm a couple more times, delete all my .Mac sync clients a couple more times, and even enable synching from iCal's & Address Book's preferences because System Preferences:.Mac:Sync wasn't usable -- it would just spin forever, never letting me actually click on the check-boxes to enable synching or specific types of data.

A week later, that part of System Preferences works again, and I have no idea why.

I also don't know how much longer it will continue to work. I was very intrigued by SyncTogether, until I discovered it just uses the same Sync Services infrastructure that keeps breaking so badly on me. I basically pay $100/year for .Mac sync, since I don't use their promo software and I run a more reliable mail server myself (on Apple hardware and software, no less!), so I'd love to find a cheaper alternative, but it's not clear if SyncTogether would really be cheaper (depends on upgrade schedule and pricing, and how many machines, and would require me to poke additional holes in my firewall). Unless and until Apple's syching gets stable, it doesn't make sense to invest more in it than I already am. Additionally, I don't know how soon Mark/Space will have Leopard support.

Wednesday, April 11 2007

Backups and Moving into Subversion

I do not have a comprehensive backup strategy. I keep reading well-reasoned arguments for at least 3 complete (bootable) backups, scratching my head, and saying, "Yeah, if I had a couple thousand unused, I could do that." Yes, I know drives are cheap, but I have 3 important systems to back up, and 4 more where perhaps a dozen files matter combined. Between the important systems, I'd need about 600gb usable, and Joe thinks you need 3 bootable backups, and I'm a strong believer in keeping the backups permanently away from the originals (not on a rotating cycle, but never-the-twain-shall-meet time -- what else is broadband for?).

http://www.takecontrolbooks.com/backup-macosx.html

Unfortunately, whenever I kick off an rsync job to update my remote backups (one set, not 3), Amy complains every day because Internet access is so slow. I think I have throttled it back enough to be unintrusive, but I just don't remember, and (especially at an unobtrusive fraction of our 384kbps DSL uplink), it just takes so dang long.

http://rsync.samba.org/

But I'm not too worried, because 95%+ of my important data is in Eudora on my iPod (yes, I know iPods die -- I've killed more than my fair share, along with Zip disks, Orb disks, and even a 2.5" FireWire drive), and I use Synchronize Plus X to back up my iPod to my primary home and work machines automatically, every time I plug it in. This is a great feature of Synchronize, which mitigates my annoyance with their licensing.

Every morning when I get to work, I dock the iPod; Synchronize sees it, launches Eudora, backs up changed files to my work desktop's hard disk, and launches an AppleScript to bring up a couple applications. When I'm done, I type eject in a Terminal window, my iPod unmounts, and login.keychain is locked (flushing ssh keys in the process). On my home PowerBook, the process is the same.

So I know my backup system works well, because I test restores whenever there's a problem with the iPod (perhaps twice a year). But I still feel nervous erasing it before sending it away. I actually did this today -- a couple months after its 2-year anniversary, my 60gb iPod photo's battery won't last an hour. Apple told me they'd fix it under extended AppleCare, but changed their minds and cancelled the repair -- without telling me. I paid the $70 and sent it in today; I'm running off our old 10gb iPod in the interim.

For the next step, I'm moving my website into Subversion. I already keep Julia's school site & Bjorn's site & documentation in Subversion, and last week I got sick of making manual backups of a piece I'm writing for TidBITS, so I reorganized and checked in my writing (not that there's much of it). Tonight, I threw out 4gb of my 7gb ~/public_html in preparation for checking in (most of) the rest of it.

Backing up a Subversion repository is easy, and will bring me to about 99% of important data backed up (not counting MP3s -- are already replicated to 2 Macs & an iPod -- or iPhoto libraries, which require different handling, but should back up quite well via rsync; but I have to get Julia's few [important] pages into Subversion too).

Monday, January 30 2006

Clearing Sync Services

iSync and .Mac Sync are great. I keep my Treo synchronized with three different Macs, and they all have the same data. Unfortunately, while Apple was enhancing the capabilities of .Mac Sync during the Tiger betas, they broke a bunch of things (including compatibility with Panther). Things have gotten better since then, but I still see problems on about a monthly basis. Symptoms are data not making it over and ".Mac login failed." errors from the .Mac System Preferences pane. Apple has a "fix", but it's pretty draconian, and Sync Services tends to get corrupted again. Here's the procedure:

  1. Disable .Mac syncing (if applicable)
  2. Close all programs. Log out and log back in.
  3. Make sure no programs are running after logging back in. If the user is [the original email I received trails off here. --cp]
  4. Create an archive of ~/Library/Application Support/SyncServices. Delete the directory after archiving it
  5. Re-enable .Mac syncing (if applicable)
  6. Sync with .Mac: go to the advanced tab, click Reset Sync Data and choose to Reset Sync Data on .Mac or on this computer depending on which one has the most recent data
  7. If you have devices, now is the time to manually synchronize them (manually add any new data from the device that is not on the computer), then do a sync with the device choosing Erase data on device then sync

I add step 0: Back up your data (iCal & Address Book have handy "Back Up" items, and Safari has "Export Bookmarks"). When logging back in, hold down the Shift key to avoid automatically launching Login Items.