My journey with rsync

A few weeks ago, I posted a blog entry about building my own backup system using rsync, instead of using MacOS’s built-in Time Machine.  I did this for a number of reasons.  Well, it’s been a few weeks and I’ve learned some lessons.  So I thought I’d share them!

Timeouts

The first thing I learned is that ssh will sometimes time out when rsyncing via ssh between two macs.  (This is probably the case between other systems.)  I’m guessing it’s when it’s building the file list from a very big drive.  To fix that problem, I edited /etc/sshd_config on each Mac.  In this scenario, the “backup server” is the ssh client and the “backup client” is the sshd server, since we are sshing from the backup server to the backup client.  Therefore, on the backup client, I edit the sshd_config file and added the value ClientAliveInterval 60.  It tells the server that if 60 seconds goes by without hearing from the client (which is what will happen during a large file transfer, or while building the file list), it should send a message via the encrypted channel and request a response from the client.  If it gets a response (which it should, since the client is alive and well) it will keep the connection alive.  This will happen over and over until things start going again. 

But since this setting is really designed to notice a dead client, I also added the ClientAliveCountMax 4 value to tell it to give up after a few tries.  (In our config, this value will never be reached, as it is a count of the number of times it sent a message to the client and it DIDN’T respond. In our config, it will respond just fine.)

Changing files

Sometimes files change or disappear during a sync.   When this happens, rsync exits with a 24 instead of a 0.  My experience so far is that it rarely happens (as I’m usually backing up at night), but it happens sometimes.  But since it happens sometimes I put in the script that 24 or 0 is OK.

Ownership

Although MacOS is based on Unix, it has the ability to ignore Unix things occasionally, like case or ownership. By default when you make a new drive in MacOS (except when you’re installing the OS), it does not enable ownership on the drive.  Meaning that no matter who writes a file to the drive, the file ends up being owned by unknown/unknown.

I use a separate backup drive for backing up my OS drives.  It’s just a SATA drive in a 3rd party enclosure.  My 250 GB drive I had in there filled up right at the time the 750 GB drive in my Drobo needed to be replaced by a 2 TB drive.  I swapped the 750 for the 250 and now I had tons of space!  Except that in a few days it filled up!  What in the world?

After investigating, I saw that each subdirectory under ./backups/ was roughly the same size.  This shouldn’t be the case.  The first one should be large, and the subsequent ones should only take up a little space each. When I looked, it was the ownership thing.  I didn’t enable ownership on the new drive, and so every file was owned by unknown.  Then when rsync would compare the new file, it looked different (because it was a different owner), and it transferred the whole file.  This also explained why the backups were taking a lot long than they used to.  Funny that.

Sleepy sleepy

When the iMac that is the backup server has a cron entry to wakeup and back everybody up, it sometimes find that all the other Macs are asleep.  Although they supposedly are supposed to have the WAKEONLAN feature, and I’ve read a few articles on how to wake them up via the LAN, I have not been able to do it.  My only solution was to create a cron job on each client to wake up just before backup time and (ironically) use the Unix sleep command to get it to stay awake until it’s appointed time.  I sure would like a more elegant solution to this. I may actually move toward having them all back themselves up to the server using their own cron jobs, cause this is downright silly.  But I’m open to suggestions.

Speed

Once the things above were worked out, it is a very fast backup method.  I backup over 4 TB of data in under 10 mins.  I also provide an offsite backup of backupcentral.com & truthinit.com by rsyncing their backup files (which are automatically created via cpanel every night) to the same drive.

Backup of BackupCentral

That takes about seven more minutes because I’m copying down hundreds of MB of files every night. Not only does this give me an offsite backup, it gives me more granularity than what cpanel offers me.  (While they do keep a monthly and weekly backup, the daily backup is overwritten every day.  So I can either restore to last night or last week. Yuck.)  So I’ve always copied it offsite every day.  This way is much more elegant and treats my backupcentral data the same way I treat me other data. One system to monitor.

Monitoring and Reporting

Now that I’ve got things completely working, I need to build some reporting into this thing.  It would be nice if I didn’t have to log in to the one iMac every day to check my status.  Of course, that means I’m going to have to figure out how to get the iMac to email me.  Excuse me while I go google some stuff.

I hope that some of you find this useful.


Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data

8 comments
  • We’ve had good success for a few years using Unison (http://www.cis.upenn.edu/~bcpierce/unison/) to keep 87GB of files in sync between a "live" FTP server and a "warm" standby server (via a 1GB Ether crossover cable).

    It has good logging, so you can review what got copied/deleted and when.

    —————-
    Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

    Unison shares a number of features with tools such as configuration management packages (CVS, PRCS, Subversion, BitKeeper, etc.), distributed filesystems (Coda, etc.), uni-directional mirroring utilities (rsync, etc.), and other synchronizers (Intellisync, Reconcile, etc). However, there are several points where it differs:

    * Unison runs on both Windows and many flavors of Unix (Solaris, Linux, OS X, etc.) systems. Moreover, Unison works across platforms, allowing you to synchronize a Windows laptop with a Unix server, for example.
    * Unlike simple mirroring or backup utilities, Unison can deal with updates to both replicas of a distributed directory structure. Updates that do not conflict are propagated automatically. Conflicting updates are detected and displayed.
    * Unlike a distributed filesystem, Unison is a user-level program: there is no need to modify the kernel or to have superuser privileges on either host.
    * Unison works between any pair of machines connected to the internet, communicating over either a direct socket link or tunneling over an encrypted ssh connection. It is careful with network bandwidth, and runs well over slow links such as PPP connections. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.
    * Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.
    * Unison has a clear and precise specification.
    * Unison is free; full source code is available under the GNU Public License.

  • I am glad to read about your experiences with rsync. A couple of years ago, I set up rsync to do nightly backups of 3 linux clients to a linux backup server. It works great and I actually enjoyed figuring it out (with the help of Google, as you mentioned). It was a nice little project for me to learn more about shell scripts, ssh, and a few unrelated things like nfs and apcupsd.

  • Curtis — +1… Thanks for a great report about your experiences with rsync. After eval and trying several open-system “solutions” to the problem of desktop backup in our SOHO environment, without having particularly demanding or complicated needs, and finding them each wanting in various aspects, I finally bit the bullet and wrote my own bash/rsync scripts.

    And although these too are wanting in certain respects, at least they’re problems/deficiencies of *my own* making, which I can either ignore or get around to repairing/improving at my own pace. Where skills/experience permits, there’s something to be said for the “roll your own” approach, especially when you’ve got a solid utility like rsync to do the work-horse parts for you. Thank you for sharing!

  • Thanks, Curtis – a useful post!

    For my Linux systems, I’ve now switched almost all my backups to duplicity (http://www.nongnu.org/duplicity/), partly because many of my backups are now on Amazon S3 which is easy and efficient with this.

    But I haven’t played with it much on a Mac yet, and I suspect some of the issues above will apply when I do!

  • P.S. I should have mentioned that duplicity is based on rsync, or at least on librsync, which is why I thought it relevant – it uses the same algorithms, but gives you full and incremental backups to a variety of destinations.

  • Good writeup. We use BackupPC, which also utilizes rsync. It has a friendly web management interface that’s useful for lower end technicians. It also has the logging and scheduling built in and was easy to set up.

    We have a backup repository at our home office and one at our remote office, both with 6 months of backups online for our 18 servers, then we back up to tape once a month so we have a full 12 months. About the only issue we had was the small pipe to our remote office, so we initialized the repository locally before we sent it down. We also do things like gzip SQL backups before we send them over to minimize traffic.

    It’s a pretty redundant solution and we barely ever have to troubleshoot any issues (mostly the tapes).

  • I experimented with running RSYNC in background daemon mode, using the RSYNC protocol.
    A ‘file share’ can be set up on the source or destination, and a ‘pull’ or ‘push’ can be scheduled.

    That is quite fast and efficient, and if you own/trust both endpoints and the network, the lack of encryption may not be of much consequence

    K

  • [quote name=Lee Roth]We’ve had good success for a few years using Unison (http://www.cis.upenn.edu/~bcpierce/unison/) to keep 87GB of files in sync between a "live" FTP server and a "warm" standby server (via a 1GB Ether crossover cable).

    It has good logging, so you can review what got copied/deleted and when.

    —————-
    Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

    Unison shares a number of features with tools such as configuration management packages (CVS, PRCS, Subversion, BitKeeper, etc.), distributed filesystems (Coda, etc.), uni-directional mirroring utilities (rsync, etc.), and other synchronizers (Intellisync, Reconcile, etc). However, there are several points where it differs:

    * Unison runs on both Windows and many flavors of Unix (Solaris, Linux, OS X, etc.) systems. Moreover, Unison works across platforms, allowing you to synchronize a Windows laptop with a Unix server, for example.
    * Unlike simple mirroring or backup utilities, Unison can deal with updates to both replicas of a distributed directory structure. Updates that do not conflict are propagated automatically. Conflicting updates are detected and displayed.
    * Unlike a distributed filesystem, Unison is a user-level program: there is no need to modify the kernel or to have superuser privileges on either host.
    * Unison works between any pair of machines connected to the internet, communicating over either a direct socket link or tunneling over an encrypted ssh connection. It is careful with network bandwidth, and runs well over slow links such as PPP connections. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.
    * Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.
    * Unison has a clear and precise specification.
    * Unison is free; full source code is available under the GNU Public License.[/quote]I was using unison for a long time, transferring about 200GB at the time, between webservers in different countries, after awhile unison developed a problem, maybe it was the dataset increasing in size, too many small files I don’t know what the problem was really the issue was I couldn’t get any support as unison seems pretty much abandoned.

    In the end I went back to rsync which has been fine for years ever since the dataset is approaching 500GB now and it still fine.