Back in August, I posted a blog entry that 10 GbE is a lie, as no one who I had talked to had ever gotten more than 400 MB/s (3200 Mb/s), and I asked anyone who had done better to write. Well, they wrote.
My new best friend is Devon Peters who posted the results of his 10 GbE testing on the NetBackup Mailing List, and he got as much as 7500 Mb/s! While that's not quite 10k Mb/s, it's pretty darn impressive. That's almost 1000 MB/s! He said that when the throughput got that high, "all the cores/threads on the receiving T2000 become saturated and TCP retransmits start climbing, but both systems remain quite responsive." Sweet!
We asked Devon how he did this, and he said it was two 1 Ghz, 4-core Sun T2000 servers hooked up back to back running Solaris 10 11/06, with the recommended patch cluster as of 9/12 (kernel patch is 125100-10). The following is a direct quote from his post:
In the end I basically bumped up the max buffer and window sizes to 10MB, enabled window scaling, and bumped up the connection queues:
The tunables that made a noticable difference regarding performance are:
nxge0 accept_jumbo 1
only one cpu/thread per core is interruptable (set using: psradm -i 1-3 5-7 9-11 13-15)
You can find Sun's recommended settings for these cards here: http://www.solarisinternals.com/wiki/index.php/Networks
Also, the iperf commands that have provided the highest throughput are:
Server: iperf -s -f m -w 512K -l 512K
Client: iperf -c <server> -f m -w 512K -l 512K -t 600 -P <numstreams>
There you go! Turns out it's not a lie after all.