Network filesystem showdown

Now that I’m trying to use my home-built NAS solution to store things like video imported from a video camera via Firewire, I needed much more performance than I was getting, as I was dropping frames during import.  NOT GOOD.  I sought counsel via Twitter, Facebook and email, and I got all kinds of conflicting advice.  I heard FreeBSD NFS was definitely faster than Linux’s NFS.  I heard AFP was definitely faster than NFS.  Then I heard that both of those were crazy.  So I decided to test for myself and see what happened.  (I do have a backup tie-in to this at the end, BTW — if you’re a MacOS customer.)
First let me describe my test environment. 

Client: Core 2 Duo iMac with 2 GHz CPU 7 3 GB of RAM and GbE (Running Snow Leopard)
OldServer: Whitebox PC AMD64 3400+ w/4 GB RAM and GbE (Running Ubuntu Desktop 9.04)  It’s storage is a USB-connected Drobo.
TestServer: Laptop w/Dual-core Intel 2 GHz & 1 GB RAM and GbE (Dual-boot Ubuntu Server 9.10 & FreeBSD 8)

I verified that all clients and servers were running at 1000 Mb and full duplex.

I had two files that I was going to copy.  The first was a small file of 66KB and the second was a 1GB VOB file from a DVD I was ripping when I started.  I wrote a script that did several tests with each file, platform, and protocol:
1. It copied the file from my iMac to the share
2. It copied the file from the share to the iMac
3. It copied the file from the share to the share
4. It renamed the file
5. It removed the file

Each test was run five times and I did my best to make sure nothing else was going on during the test.  (This is a “production” network after all.)

First let’s talk about the CIFS server. The performance was actually so poor that I checked to make sure it wasn’t the USB-connected Drobo causing the problem.  (It’s a Generation 1 Drobo. The newer ones are much faster.)  So I ran the test against local storage on the system itself. No, the Drobo was not the issue.

CIFS results: It took about two minutes to copy the 1 GB file from the iMac to or from the CIFS share, and almost FOUR minutes (215 seconds) to copy it from the share to the share.  That must be why I was dropping frames.  OK.

I then ran an NFS bake-off between FreeBSD and Linux.  I am very unfamiliar with FreeBSD (never installed it, actually) and it is very different than its Linux counterpart. After some help from some bloggers, I managed to get the laptop to dual-boot Linux & FreeBSD.  Everything was kept standard; no custom kernels were created and I did no NFS tweaking.  (The test was taking enough of my time.)

NFS Results: The results were pretty much a dead heat.  50 seconds for the write, 95 seconds for the read, and 55 seconds for the read/write.  So much for FreeBSD’s superior NFS performance.  (A friend said that this was true a long time ago when Linux’s NFS ran in user-space.  Yuck.)  That worked for me because there are only so many new OSs a guy can learn in a quarter.  (I switched to Macs and have Windows 7 running in a VM.  I’m not “at home” anywhere right now.)

Given the dead heat on NFS, and the lack of clear documentation on AFP on FreeBSD, I decided to configure AFP on Linux only.  Having never touched AFP, I did some searches. I learned about something called netatalk, which provides AFP services on Unix systems.  I again found all kinds of different answers as to how to do it.  Then I read someone that in Ubuntu 9.10, all I had to do was “apt-get install netatalk.”  I did and it just worked!  WOW!  NFS I can handle, because I know it, but it still takes a few steps to get going.  Samba can be a bear to configure, but there is lots of help.  But with netatalk on Ubuntu 9.10, all I did was install it and voila!  All my home directories were automatically shared.  So I ran the test against AFP, kind of hoping it would be a dead heat with NFS.

AFP Results: It left NFS in the DUST!  25 seconds on writes, 23 seconds on reads, and 43 seconds on read/write. 

So there you have it, folks.  AFP is officially faster (almost three times faster in some cases) than NFS.

I know this doesn’t simulate real-world performance, with multiple people hitting at the same time, etc, but real-time performance was not my problem.  Importing and exporting video was my problem.  That required throughput.

If there’s anyone out there that can tell me how to tune NFS to performance better, feel free to send me an email.  But it looks like I’ll be going with AFP.

Are you wondering about the backup angle?  By using AFP, I can also use my AFP server as a target for Time Machine backups.  How about that.

For those interested in the raw numbers, here they are:

FreeBSD 8 NFS:1GB Write:         48.93 real         0.00 user        13.74 sys
FreeBSD 8 NFS:1GB Write:         50.22 real         0.00 user        13.61 sys
FreeBSD 8 NFS:1GB Write:         48.67 real         0.00 user        13.53 sys
FreeBSD 8 NFS:1GB Write:         50.37 real         0.00 user        13.66 sys
FreeBSD 8 NFS:1GB Write:         50.16 real         0.00 user        13.84 sys
Ubuntu 9.10 AFP:1GB Write:       24.59 real         0.00 user        11.02 sys
Ubuntu 9.10 AFP:1GB Write:       24.08 real         0.00 user        11.14 sys
Ubuntu 9.10 AFP:1GB Write:       23.09 real         0.00 user        11.17 sys
Ubuntu 9.10 AFP:1GB Write:       25.76 real         0.00 user        11.19 sys
Ubuntu 9.10 AFP:1GB Write:       23.96 real         0.00 user        11.40 sys
Ubuntu 9.04 CIFS:1GB Write:      112.04 real        0.00 user         3.16 sys
Ubuntu 9.04 CIFS:1GB Write:      112.55 real        0.00 user         3.33 sys
Ubuntu 9.04 CIFS:1GB Write:      112.63 real        0.00 user         3.36 sys
Ubuntu 9.04 CIFS:1GB Write:      112.36 real        0.00 user         3.14 sys
Ubuntu 9.04 CIFS:1GB Write:      112.23 real        0.00 user         3.14 sys
Ubuntu 9.10 NFS:1GB Write:       53.80 real         0.00 user        14.01 sys
Ubuntu 9.10 NFS:1GB Write:       53.54 real         0.00 user        14.01 sys
Ubuntu 9.10 NFS:1GB Write:       54.77 real         0.00 user        14.11 sys
Ubuntu 9.10 NFS:1GB Write:       53.72 real         0.00 user        14.05 sys
Ubuntu 9.10 NFS:1GB Write:       53.82 real         0.00 user        14.16 sys

FreeBSD 8 NFS:1GB Read:         99.38 real         0.00 user         4.48 sys
FreeBSD 8 NFS:1GB Read:         93.39 real         0.00 user         4.34 sys
FreeBSD 8 NFS:1GB Read:         93.73 real         0.00 user         4.37 sys
FreeBSD 8 NFS:1GB Read:         97.48 real         0.00 user         4.41 sys
FreeBSD 8 NFS:1GB Read:         97.53 real         0.00 user         4.45 sys
Ubuntu 9.10 AFP:1GB Read:       23.00 real         0.00 user         3.27 sys
Ubuntu 9.10 AFP:1GB Read:       23.33 real         0.00 user         3.26 sys
Ubuntu 9.10 AFP:1GB Read:       24.04 real         0.00 user         3.32 sys
Ubuntu 9.10 AFP:1GB Read:       28.26 real         0.00 user         3.43 sys
Ubuntu 9.10 AFP:1GB Read:       35.80 real         0.00 user         3.84 sys
Ubuntu 9.04 CIFS:1GB Read:      119.02 real        0.00 user         3.08 sys
Ubuntu 9.04 CIFS:1GB Read:      118.39 real        0.00 user         3.45 sys
Ubuntu 9.04 CIFS:1GB Read:      119.07 real        0.00 user         3.26 sys
Ubuntu 9.04 CIFS:1GB Read:      117.40 real        0.00 user         3.04 sys
Ubuntu 9.04 CIFS:1GB Read:      118.04 real        0.00 user         3.05 sys
Ubuntu 9.10 NFS:1GB Read:       94.87 real         0.00 user         4.43 sys
Ubuntu 9.10 NFS:1GB Read:       96.19 real         0.00 user         4.42 sys
Ubuntu 9.10 NFS:1GB Read:       99.62 real         0.00 user         4.37 sys
Ubuntu 9.10 NFS:1GB Read:       94.73 real         0.00 user         4.35 sys
Ubuntu 9.10 NFS:1GB Read:       96.35 real         0.00 user         4.42 sys

FreeBSD 8 NFS:1GB R/W:         59.52 real         0.00 user        13.76 sys
FreeBSD 8 NFS:1GB R/W:         52.08 real         0.00 user        13.65 sys
FreeBSD 8 NFS:1GB R/W:         63.96 real         0.00 user        13.86 sys
FreeBSD 8 NFS:1GB R/W:         49.28 real         0.00 user        13.91 sys
FreeBSD 8 NFS:1GB R/W:         51.67 real         0.00 user        13.92 sys
Ubuntu 9.10 AFP:1GB R/W:       41.17 real         0.00 user        12.06 sys
Ubuntu 9.10 AFP:1GB R/W:       43.01 real         0.00 user        12.11 sys
Ubuntu 9.10 AFP:1GB R/W:       39.02 real         0.00 user        12.21 sys
Ubuntu 9.10 AFP:1GB R/W:       48.89 real         0.00 user        12.56 sys
Ubuntu 9.10 AFP:1GB R/W:       53.27 real         0.00 user        13.16 sys
Ubuntu 9.04 CIFS:1GB R/W:      217.32 real        0.00 user         4.34 sys
Ubuntu 9.04 CIFS:1GB R/W:      217.55 real        0.00 user         4.91 sys
Ubuntu 9.04 CIFS:1GB R/W:      217.07 real        0.00 user         4.50 sys
Ubuntu 9.04 CIFS:1GB R/W:      217.67 real        0.00 user         4.71 sys
Ubuntu 9.04 CIFS:1GB R/W:      217.23 real        0.00 user         4.97 sys
Ubuntu 9.10 NFS:1GB R/W:       53.48 real         0.00 user        14.12 sys
Ubuntu 9.10 NFS:1GB R/W:       58.50 real         0.00 user        13.97 sys
Ubuntu 9.10 NFS:1GB R/W:       53.79 real         0.00 user        14.19 sys
Ubuntu 9.10 NFS:1GB R/W:       53.01 real         0.00 user        14.16 sys
Ubuntu 9.10 NFS:1GB R/W:       53.38 real         0.00 user        14.15 sys

And here’s the script I wrote to do the test.  Yes I know about iozone, but it requires the Darwin ports on Mac OS, and I don’t have them installed yet.  I plan to repeat the test once that happens.

#!/bin/sh

if [ $# -lt 1 ] ; then
  echo “Enter network directory name: “
  read a
else
  a=$1
fi

if [ $# -eq 3 ] ; then
  os=$2
else
  echo “Please enter l for Linux or f for FreeBSD”
  read os
fi

if [ $# -eq 3 ] ; then
  fs=$3
else
  echo “Please enter n for nfs or a for afp”
  read fs
fi

if [ “$os” != “l” ] ; then
  if [ “$os” != “f” ] ; then
  echo “Usage: $0 [directory] [os] [fs]”
  exit
  fi
fi

if [ “$fs” != “n” ] ; then
  if [ “$fs” != “a” ] ; then
    if [ “$fs” != “c” ] ; then
      echo “Usage: $0 [directory] [l|f] [n|a|c]”
      echo “(where l=linux, f=freebsd, n=nfs, a=afp, c=cifs)”
      exit
    fi
  fi
fi

echo

if [ -d “$a” ] ; then
  ok=1
else
  echo “Unfortunately, “$a” does not appear to be a directory”
  echo “Exiting”
  exit
fi

touch “$a/file”
if [ $? -gt 0 ] ; then
  echo “Cannot write to directory “$a””
  echo “Exiting”  exit 1
else  rm -f “$a/file”
fi
cp /dev/null logfile.$os.$fs
for i in 1 2 3 4 5do
rm -f *.2 2>/dev/null
rm -f “$a/*” 2>/dev/null
echo “Testing small files”
/usr/bin/time cp smallfile “$a/smallfile” 2>&1 |sed “s/^/$os:$fs:scpton:/” >>logfile
.$os.$fs
#/usr/bin/time cat smallfile >> “$a/smallfile” 2>&1 |sed “s/^/$os:$fs:sapp:/” >>logf
ile.$os.$fs
/usr/bin/time cp “$a/smallfile” smallfile.2 2>&1 |sed “s/^/$os:$fs:scpfrom:/” >>logfile.$os.$fs
/usr/bin/time cp “$a/smallfile” /dev/null 2>&1 |sed “s/^/$os:$fs:scptonull:/” >>logf
ile.$os.$fs/usr/bin/time cp “$a/smallfile” “$a/smallfile.2” 2>&1 |sed “s/^/$os:$fs:scpin:/” >>l
ogfile.$os.$fs
/usr/bin/time mv “$a/smallfile” “$a/smallfile.moved” 2>&1 |sed “s/^/$os:$fs:smv1:/” >>logfile.$os.$fs
/usr/bin/time mv “$a/smallfile.moved” “$a/smallfile” 2>&1 |sed “s/^/$os:$fs:smv2:/”
>>logfile.$os.$fs/usr/bin/time rm -f “$a/smallfile” 2>&1 |sed “s/^/$os:$fs:srm:/” >>logfile.$os.$fs
/usr/bin/time rm -f “$a/smallfile.2” 2>&1 |sed “s/^/$os:$fs:srm:/” >>logfile.$os.$fs
echo “Testing big files”
echo cpto
/usr/bin/time cp bigfile “$a/bigfile” 2>&1 |sed “s/^/$os:$fs:bcpton:/” >>logfile.$os.$fs
echo app
#/usr/bin/time cat smallfile >> “$a/bigfile” 2>&1 |sed “s/^/$os:$fs:bapp:/” >>logfile.$os.$fs
echo cprom
/usr/bin/time cp “$a/bigfile” bigfile.2 2>&1 |sed “s/^/$os:$fs:bcpfrom:/” >>logfile.
$os.$fs
echo cpnull
/usr/bin/time cp “$a/bigfile” /dev/null 2>&1 |sed “s/^/$os:$fs:bcptonull:/” >>logfil
e.$os.$fs
echo cpnfs2nfs
/usr/bin/time cp “$a/bigfile” “$a/bigfile.2” 2>&1 |sed “s/^/$os:$fs:bcpin:/” >>logfile.$os.$fs
echo mv1
/usr/bin/time mv “$a/bigfile” “$a/bigfile.moved” 2>&1 |sed “s/^/$os:$fs:bmv1:/” >>logfile.$os.$fs
echo mv2
/usr/bin/time mv “$a/bigfile.moved” “$a/bigfile” 2>&1 |sed “s/^/$os:$fs:bmv2:/” >>logfile.$os.$fs
echo rm1
/usr/bin/time rm -f “$a/bigfile” 2>&1 |sed “s/^/$os:$fs:brm:/” >>logfile.$os.$fs
echo rm2
/usr/bin/time rm -f “$a/bigfile.2” 2>&1 |sed “s/^/$os:$fs:brm:/” >>logfile.$os.$fs

done


Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data