10 GbE is real!

Back in August, I posted a blog entry that 10 GbE is a lie, as no one who I had talked to had ever gotten more than 400 MB/s (3200 Mb/s), and I asked anyone who had done better to write.  Well, they wrote.

My new best friend is Devon Peters who posted the results of his 10 GbE testing on the NetBackup Mailing List, and he got as much as 7500 Mb/s!  While that's not quite 10k Mb/s, it's pretty darn impressive.  That's almost 1000 MB/s!  He said that when the throughput got that high, "all the cores/threads on the receiving T2000 become saturated and TCP retransmits start climbing, but both systems remain quite responsive."  Sweet!

We asked Devon how he did this, and he said it was two 1 Ghz, 4-core Sun T2000 servers hooked up back to back running Solaris 10 11/06, with the recommended patch cluster as of 9/12 (kernel patch is 125100-10).  The following is a direct quote from his post: 

In the end I basically bumped up the max buffer and window sizes to 10MB, enabled window scaling, and bumped up the connection queues:

tcp_conn_req_max_q 8192
tcp_conn_req_max_q0 8192
tcp_max_buf 10485760
tcp_cwnd_max 10485760
tcp_recv_hiwat 65536
tcp_xmit_hiwat 65536

The tunables that made a noticable difference regarding performance are:

ddi_msix_alloc_limit 8
tcp_squeue_wput 1
ip_soft_rings_cnt 64
ip_squeue_fanout 1
nxge0 accept_jumbo 1

only one cpu/thread per core is interruptable (set using: psradm -i 1-3 5-7 9-11 13-15)

You can find Sun's recommended settings for these cards here: http://www.solarisinternals.com/wiki/index.php/Networks

Also, the iperf commands that have provided the highest throughput are:

Server: iperf -s -f m -w 512K -l 512K
Client: iperf -c <server> -f m -w 512K -l 512K -t 600 -P <numstreams> 

There you go! Turns out it's not a lie after all.

Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at Sullivan Strickler, which helps companies manage their legacy data