| View previous topic :: View next topic |
| Author |
Message |
devon
Joined: 24 Aug 2007
Posts: 17
|
 Posted: Wed Oct 17, 2007 11:15 am Post subject: Some info on my experiences with 10GbE |
 |
|
Since I've seen a little bit of talk about 10GbE on here in the past I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on some small T2000's (1Ghz, 4-core). I'm only using a single port on each card, and the servers are currently directly connected to each other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems to about 7500Mbit/sec using iperf. When the throughput gets this high, all the cores/threads on the receiving T2000 become saturated and TCP retransmits start climbing, but both systems remain quite responsive. Since these are only 4-core T2000's, I would guess that the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires lots of data streams. When transmitting with a single data stream, the most throughput I've gotten is about 1500Mbit/sec. I only got up to 7500Mbit/s when using 64 data streams… Also, the biggest gains seem to be in the jump from 1 to 8 data streams; with 8 streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a speed of at least 2400Mbit/sec (300MB/sec). We have large daily backups (3-4TB) that we would like to be able to restore (not backup) in a reasonable amount of time. These restores are used to refresh our test and development environments with current data. The actual backups are done with array based snapshots (HDS ShadowCopy), which then get mounted and backed up by a dedicated media server (6-core T2000). We're currently getting about 650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams should get us well over 2400Mbit/sec. But - we haven't installed the cards on our media servers yet, so I have yet to see what the actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful it'll be close to the iperf results, but if it doesn't meet the goal then we'll be looking at other options.
--
Devon Peters |
|
| Back to top |
|
 |
jpiszcz
Joined: 02 Sep 2007
Posts: 371
|
| Posted: Wed Oct 17, 2007 12:33 pm Post subject: Some info on my experiences with 10GbE |
|
|
Very nice write up and useful information, thanks!
On Wed, 17 Oct 2007, Peters, Devon C wrote:
| Quote: | Since I've seen a little bit of talk about 10GbE on here in the past I
figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on some
small T2000's (1Ghz, 4-core). I'm only using a single port on each
card, and the servers are currently directly connected to each other
(waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems to
about 7500Mbit/sec using iperf. When the throughput gets this high, all
the cores/threads on the receiving T2000 become saturated and TCP
retransmits start climbing, but both systems remain quite responsive.
Since these are only 4-core T2000's, I would guess that the 6 or 8-core
T2000's (especially with 1.2Ghz or 1.4Ghz processors) should be capable
of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires lots
of data streams. When transmitting with a single data stream, the most
throughput I've gotten is about 1500Mbit/sec. I only got up to
7500Mbit/s when using 64 data streams... Also, the biggest gains seem
to be in the jump from 1 to 8 data streams; with 8 streams I was able
to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a speed
of at least 2400Mbit/sec (300MB/sec). We have large daily backups
(3-4TB) that we would like to be able to restore (not backup) in a
reasonable amount of time. These restores are used to refresh our test
and development environments with current data. The actual backups are
done with array based snapshots (HDS ShadowCopy), which then get mounted
and backed up by a dedicated media server (6-core T2000). We're
currently getting about 650MB/sec of throughput with the backups (9
streams on 3 LTO3 tape drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams
should get us well over 2400Mbit/sec. But - we haven't installed the
cards on our media servers yet, so I have yet to see what the actual
performanee of netbackup and LTO3 over 10GbE is. I'm hopeful it'll be
close to the iperf results, but if it doesn't meet the goal then we'll
be looking at other options.
--
Devon Peters
|
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
cpreston
Site Admin
Joined: 04 May 2007
Posts: 556
|
| Posted: Thu Oct 18, 2007 12:44 am Post subject: Some info on my experiences with 10GbE |
|
|
7500 MB/s! That’s the most impressive numbers I’ve ever seen by FAR. I may have to take back my “10 GbE is a Lie!” blog post, and I’d be happy to do so.
Can you share things besides the T2000? For example,
what OS and patch levels are you running?
Any IP patches?
Any IP-specific patches?
What ndd settings are you using?
Is rss enabled?
“Input, I need Input!”
---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies
From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on some small T2000's (1Ghz, 4-core). I'm only using a single port on each card, and the servers are currently directly connected to each other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems to about 7500Mbit/sec using iperf. When the throughput gets this high, all the cores/threads on the receiving T2000 become saturated and TCP retransmits start climbing, but both systems remain quite responsive. Since these are only 4-core T2000's, I would guess that the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires lots of data streams. When transmitting with a single data stream, the most throughput I've gotten is about 1500Mbit/sec. I only got up to 7500Mbit/s when using 64 data streams… Also, the biggest gains seem to be in the jump from 1 to 8 data streams; with 8 streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a speed of at least 2400Mbit/sec (300MB/sec). We have large daily backups (3-4TB) that we would like to be able to restore (not backup) in a reasonable amount of time. These restores are used to refresh our test and development environments with current data. The actual backups are done with array based snapshots (HDS ShadowCopy), which then get mounted and backed up by a dedicated media server (6-core T2000). We're currently getting about 650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams should get us well over 2400Mbit/sec. But - we haven't installed the cards on our media servers yet, so I have yet to see what the actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful it'll be close to the iperf results, but if it doesn't meet the goal then we'll be looking at other options.
--
Devon Peters |
|
| Back to top |
|
|
amellor.au
Joined: 17 May 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 1:36 am Post subject: Some info on my experiences with 10GbE |
|
|
Also, Very Interested,
My backup servers are 8core T2000's, we will be putting the SUN nxge 10Gbit cards into them.
I have only seen poor results from the card in back to back configuration (the network is not at 10Gbit yet). so far i have been misserable with results in line with Mr Preston's.
More that happy (Change controll permitting) to duplicate your setup with the 8 core boxes.
Adam.
From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Curtis Preston
Sent: Thursday, 18 October 2007 4:07 PM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
7500 MB/s! That’s the most impressive numbers I’ve ever seen by FAR. I may have to take back my “10 GbE is a Lie!” blog post, and I’d be happy to do so.
Can you share things besides the T2000? For example,
what OS and patch levels are you running?
Any IP patches?
Any IP-specific patches?
What ndd settings are you using?
Is rss enabled?
“Input, I need Input!”
---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies
From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on some small T2000's (1Ghz, 4-core). I'm only using a single port on each card, and the servers are currently directly connected to each other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems to about 7500Mbit/sec using iperf. When the throughput gets this high, all the cores/threads on the receiving T2000 become saturated and TCP retransmits start climbing, but both systems remain quite responsive. Since these are only 4-core T2000's, I would guess that the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires lots of data streams. When transmitting with a single data stream, the most throughput I've gotten is about 1500Mbit/sec. I only got up to 7500Mbit/s when using 64 data streams… Also, the biggest gains seem to be in the jump from 1 to 8 data streams; with 8 streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a speed of at least 2400Mbit/sec (300MB/sec). We have large daily backups (3-4TB) that we would like to be able to restore (not backup) in a reasonable amount of time. These restores are used to refresh our test and development environments with current data. The actual backups are done with array based snapshots (HDS ShadowCopy), which then get mounted and backed up by a dedicated media server (6-core T2000). We're currently getting about 650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams should get us well over 2400Mbit/sec. But - we haven't installed the cards on our media servers yet, so I have yet to see what the actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful it'll be close to the iperf results, but if it doesn't meet the goal then we'll be looking at other options.
--
Devon Peters
| Quote: |
NOTICE: This email and any attachments are confidential.
They may contain legally privileged information or
copyright material. You must not read, copy, use or
disclose them without authorisation. If you are not an
intended recipient, please contact us at once by return
email and then delete both messages and all attachments.
|
|
|
| Back to top |
|
|
Karl.Rossing
Guest
|
| Posted: Thu Oct 18, 2007 5:15 am Post subject: Some info on my experiences with 10GbE |
|
|
Devon,
Good to hear that T2000's are screamers.
What are the library/tape drive specs. Are the drives FC attached? or are they attached via scsi to the media server?
Thanks,
Karl
| Quote: | From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-
bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past
I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on
some small T2000's (1Ghz, 4-core). I'm only using a single port on
each card, and the servers are currently directly connected to each
other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems
to about 7500Mbit/sec using iperf. When the throughput gets this
high, all the cores/threads on the receiving T2000 become saturated
and TCP retransmits start climbing, but both systems remain quite
responsive. Since these are only 4-core T2000's, I would guess that
the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz
processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires
lots of data streams. When transmitting with a single data stream,
the most throughput I've gotten is about 1500Mbit/sec. I only got
up to 7500Mbit/s when using 64 data streams… Also, the biggest
gains seem to be in the jump from 1 to 8 data streams; with 8
streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a
speed of at least 2400Mbit/sec (300MB/sec). We have large daily
backups (3-4TB) that we would like to be able to restore (not
backup) in a reasonable amount of time. These restores are used to
refresh our test and development environments with current data.
The actual backups are done with array based snapshots (HDS
ShadowCopy), which then get mounted and backed up by a dedicated
media server (6-core T2000). We're currently getting about
650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape
drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams
should get us well over 2400Mbit/sec. But - we haven't installed
the cards on our media servers yet, so I have yet to see what the
actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful
it'll be close to the iperf results, but if it doesn't meet the goal
then we'll be looking at other options.
--
Devon Peters _______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
|
|
| Back to top |
|
|
devon
Joined: 24 Aug 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 10:41 am Post subject: Some info on my experiences with 10GbE |
|
|
I'd be be glad to share...
The OS is sol10 11/06, and I'm running the recommended patch cluster that was available on 9/12 - kernel patch is 125100-10.
For tunables, I've tested quite a few different permutations of settings for tcp, but I didn't find a whole lot to be gained from this. Peformance seemed to be best, as long as I was using a TCP congestion window of 512k or 1024k (sol10 default max is 1024k). In the end I basically bumped up the max buffer and window sizes to 10MB, enabled window scaling, and bumped up the connection queues:
tcp_conn_req_max_q 8192
tcp_conn_req_max_q0 8192
tcp_max_buf 10485760
tcp_cwnd_max 10485760
tcp_recv_hiwat 65536
tcp_xmit_hiwat 65536
The tunables that made a noticable difference regarding performance are:
ddi_msix_alloc_limit 8
tcp_squeue_wput 1
ip_soft_rings_cnt 64
ip_squeue_fanout 1
nxge0 accept_jumbo 1
only one cpu/thread per core is interruptable (set using: psradm -i 1-3 5-7 9-11 13-15)
You can find Sun's recommended settings for these cards here: http://www.solarisinternals.com/wiki/index.php/Networks
Also, the iperf commands that have provided the highest throughput are:
Server: iperf -s -f m -w 512K -l 512K
Client: iperf -c <server> -f m -w 512K -l 512K -t 600 -P <numstreams>
"Is rss enabled?" Not sure what you're asking here...
-devon
From: Curtis Preston [mailto:cpreston < at > glasshouse.com]
Sent: Thursday, October 18, 2007 1:07 AM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: RE: [Veritas-bu] Some info on my experiences with 10GbE
7500 MB/s! That’s the most impressive numbers I’ve ever seen by FAR. I may have to take back my “10 GbE is a Lie!” blog post, and I’d be happy to do so.
Can you share things besides the T2000? For example,
what OS and patch levels are you running?
Any IP patches?
Any IP-specific patches?
What ndd settings are you using?
Is rss enabled?
“Input, I need Input!”
---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies
From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on some small T2000's (1Ghz, 4-core). I'm only using a single port on each card, and the servers are currently directly connected to each other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems to about 7500Mbit/sec using iperf. When the throughput gets this high, all the cores/threads on the receiving T2000 become saturated and TCP retransmits start climbing, but both systems remain quite responsive. Since these are only 4-core T2000's, I would guess that the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires lots of data streams. When transmitting with a single data stream, the most throughput I've gotten is about 1500Mbit/sec. I only got up to 7500Mbit/s when using 64 data streams… Also, the biggest gains seem to be in the jump from 1 to 8 data streams; with 8 streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a speed of at least 2400Mbit/sec (300MB/sec). We have large daily backups (3-4TB) that we would like to be able to restore (not backup) in a reasonable amount of time. These restores are used to refresh our test and development environments with current data. The actual backups are done with array based snapshots (HDS ShadowCopy), which then get mounted and backed up by a dedicated media server (6-core T2000). We're currently getting about 650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams should get us well over 2400Mbit/sec. But - we haven't installed the cards on our media servers yet, so I have yet to see what the actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful it'll be close to the iperf results, but if it doesn't meet the goal then we'll be looking at other options.
--
Devon Peters |
|
| Back to top |
|
|
devon
Joined: 24 Aug 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 11:12 am Post subject: Some info on my experiences with 10GbE |
|
|
We've been pretty happy with the T2000's.
The tape library is an IBM 3584, the tape drives are IBM's 4Gb FC LTO-3 drives, there's a dedicated 4Gb HBA for each drive, and everything is connected to 4Gb McData switches.
We used to have IBM's 2Gb FC LTO-3 drives, and with those the peak performance was around 165MB/s per drive. These 4Gb drives peak at around 265MB/s per drive, though with all 3 tape drives active, we see throughput closer to 220MB/s per drive...I'm guessing we're bottlenecked by the ports on our disk subsystem at the moment, but since performance is more than acceptable we're not looking to tune this any further - at least not until our LTO-4 drives are installed next month  .
-devon
From: Karl.Rossing < at > Federated.CA [mailto:Karl.Rossing < at > Federated.CA]
Sent: Thursday, October 18, 2007 6:10 AM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
Devon,
Good to hear that T2000's are screamers.
What are the library/tape drive specs. Are the drives FC attached? or are they attached via scsi to the media server?
Thanks,
Karl
| Quote: | From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-
bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past
I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on
some small T2000's (1Ghz, 4-core). I'm only using a single port on
each card, and the servers are currently directly connected to each
other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems
to about 7500Mbit/sec using iperf. When the throughput gets this
high, all the cores/threads on the receiving T2000 become saturated
and TCP retransmits start climbing, but both systems remain quite
responsive. Since these are only 4-core T2000's, I would guess that
the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz
processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires
lots of data streams. When transmitting with a single data stream,
the most throughput I've gotten is about 1500Mbit/sec. I only got
up to 7500Mbit/s when using 64 data streams… Also, the biggest
gains seem to be in the jump from 1 to 8 data streams; with 8
streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a
speed of at least 2400Mbit/sec (300MB/sec). We have large daily
backups (3-4TB) that we would like to be able to restore (not
backup) in a reasonable amount of time. These restores are used to
refresh our test and development environments with current data.
The actual backups are done with array based snapshots (HDS
ShadowCopy), which then get mounted and backed up by a dedicated
media server (6-core T2000). We're currently getting about
650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape
drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams
should get us well over 2400Mbit/sec. But - we haven't installed
the cards on our media servers yet, so I have yet to see what the
actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful
it'll be close to the iperf results, but if it doesn't meet the goal
then we'll be looking at other options.
--
Devon Peters _______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
|
|
| Back to top |
|
|
Hall, Christian N.
Guest
|
| Posted: Thu Oct 18, 2007 11:24 am Post subject: Some info on my experiences with 10GbE |
|
|
Devon,
What is your data type your backing up? How much data?
Thanks,
Chris Hall
From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Thursday, October 18, 2007 3:10 PM
To: Karl.Rossing < at > federated.ca; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
We've been pretty happy with the T2000's.
The tape library is an IBM 3584, the tape drives are IBM's 4Gb FC LTO-3 drives, there's a dedicated 4Gb HBA for each drive, and everything is connected to 4Gb McData switches.
We used to have IBM's 2Gb FC LTO-3 drives, and with those the peak performance was around 165MB/s per drive. These 4Gb drives peak at around 265MB/s per drive, though with all 3 tape drives active, we see throughput closer to 220MB/s per drive...I'm guessing we're bottlenecked by the ports on our disk subsystem at the moment, but since performance is more than acceptable we're not looking to tune this any further - at least not until our LTO-4 drives are installed next month .
-devon
From: Karl.Rossing < at > Federated.CA [mailto:Karl.Rossing < at > Federated.CA]
Sent: Thursday, October 18, 2007 6:10 AM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
Devon,
Good to hear that T2000's are screamers.
What are the library/tape drive specs. Are the drives FC attached? or are they attached via scsi to the media server?
Thanks,
Karl
| Quote: | From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-
bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past
I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on
some small T2000's (1Ghz, 4-core). I'm only using a single port on
each card, and the servers are currently directly connected to each
other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems
to about 7500Mbit/sec using iperf. When the throughput gets this
high, all the cores/threads on the receiving T2000 become saturated
and TCP retransmits start climbing, but both systems remain quite
responsive. Since these are only 4-core T2000's, I would guess that
the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz
processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires
lots of data streams. When transmitting with a single data stream,
the most throughput I've gotten is about 1500Mbit/sec. I only got
up to 7500Mbit/s when using 64 data streams… Also, the biggest
gains seem to be in the jump from 1 to 8 data streams; with 8
streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a
speed of at least 2400Mbit/sec (300MB/sec). We have large daily
backups (3-4TB) that we would like to be able to restore (not
backup) in a reasonable amount of time. These restores are used to
refresh our test and development environments with current data.
The actual backups are done with array based snapshots (HDS
ShadowCopy), which then get mounted and backed up by a dedicated
media server (6-core T2000). We're currently getting about
650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape
drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams
should get us well over 2400Mbit/sec. But - we haven't installed
the cards on our media servers yet, so I have yet to see what the
actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful
it'll be close to the iperf results, but if it doesn't meet the goal
then we'll be looking at other options.
--
Devon Peters _______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
|
|
| Back to top |
|
|
cpreston
Site Admin
Joined: 04 May 2007
Posts: 556
|
| Posted: Thu Oct 18, 2007 11:49 am Post subject: Some info on my experiences with 10GbE |
|
|
"Is rss enabled?" Not sure what you're asking here...
RSS is receive-side scaling, which apparently helps improve performance:
http://www.microsoft.com/whdc/device/network/ndis_rss.mspx
I actually just learned about it the other day talking to a 10 GbE NIC vendor. He asked me that question about another user that had posted different results (but still on Solaris), and so I thought I'd ask you. Upon further research, it appears to be a Microsoft-only thing.
So, never mind!
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
devon
Joined: 24 Aug 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 12:05 pm Post subject: Some info on my experiences with 10GbE |
|
|
That sounds pretty similar to what ip_squeue_fanout does for Solaris10 - and using it made a noticable performance improvement:
ip_squeue_fanout
Description
Determines the mode of associating TCP/IP connections with squeues
A value of 0 associates a new TCP/IP connection with the CPU that creates the connection. A value of 1 associates the connection with multiple squeues that belong to different CPUs. The number of squeues that are used to fanout the connection is based upon ip_soft_rings_cnt.
-devon
-----Original Message-----
From: Curtis Preston [mailto:cpreston < at > glasshouse.com]
Sent: Thursday, October 18, 2007 12:47 PM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: RE: [Veritas-bu] Some info on my experiences with 10GbE
"Is rss enabled?" Not sure what you're asking here...
RSS is receive-side scaling, which apparently helps improve performance:
http://www.microsoft.com/whdc/device/network/ndis_rss.mspx
I actually just learned about it the other day talking to a 10 GbE NIC vendor. He asked me that question about another user that had posted different results (but still on Solaris), and so I thought I'd ask you. Upon further research, it appears to be a Microsoft-only thing.
So, never mind!
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
devon
Joined: 24 Aug 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 12:10 pm Post subject: Some info on my experiences with 10GbE |
|
|
The data is oracle database files and archive logs, and they compress real well. The largest single database is about 4TB.
-devon
From: Hall, Christian N. [mailto:HallC < at > SEC.GOV]
Sent: Thursday, October 18, 2007 12:22 PM
To: Peters, Devon C; Karl.Rossing < at > federated.ca; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: RE: [Veritas-bu] Some info on my experiences with 10GbE
Devon,
What is your data type your backing up? How much data?
Thanks,
Chris Hall
From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Thursday, October 18, 2007 3:10 PM
To: Karl.Rossing < at > federated.ca; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
We've been pretty happy with the T2000's.
The tape library is an IBM 3584, the tape drives are IBM's 4Gb FC LTO-3 drives, there's a dedicated 4Gb HBA for each drive, and everything is connected to 4Gb McData switches.
We used to have IBM's 2Gb FC LTO-3 drives, and with those the peak performance was around 165MB/s per drive. These 4Gb drives peak at around 265MB/s per drive, though with all 3 tape drives active, we see throughput closer to 220MB/s per drive...I'm guessing we're bottlenecked by the ports on our disk subsystem at the moment, but since performance is more than acceptable we're not looking to tune this any further - at least not until our LTO-4 drives are installed next month .
-devon
From: Karl.Rossing < at > Federated.CA [mailto:Karl.Rossing < at > Federated.CA]
Sent: Thursday, October 18, 2007 6:10 AM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
Devon,
Good to hear that T2000's are screamers.
What are the library/tape drive specs. Are the drives FC attached? or are they attached via scsi to the media server?
Thanks,
Karl
| Quote: | From: veritas-bu-bounces < at > mailman.eng.auburn.edu [mailto:veritas-bu-
bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters, Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past
I figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on
some small T2000's (1Ghz, 4-core). I'm only using a single port on
each card, and the servers are currently directly connected to each
other (waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems
to about 7500Mbit/sec using iperf. When the throughput gets this
high, all the cores/threads on the receiving T2000 become saturated
and TCP retransmits start climbing, but both systems remain quite
responsive. Since these are only 4-core T2000's, I would guess that
the 6 or 8-core T2000's (especially with 1.2Ghz or 1.4Ghz
processors) should be capable of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires
lots of data streams. When transmitting with a single data stream,
the most throughput I've gotten is about 1500Mbit/sec. I only got
up to 7500Mbit/s when using 64 data streams… Also, the biggest
gains seem to be in the jump from 1 to 8 data streams; with 8
streams I was able to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a
speed of at least 2400Mbit/sec (300MB/sec). We have large daily
backups (3-4TB) that we would like to be able to restore (not
backup) in a reasonable amount of time. These restores are used to
refresh our test and development environments with current data.
The actual backups are done with array based snapshots (HDS
ShadowCopy), which then get mounted and backed up by a dedicated
media server (6-core T2000). We're currently getting about
650MB/sec of throughput with the backups (9 streams on 3 LTO3 tape
drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams
should get us well over 2400Mbit/sec. But - we haven't installed
the cards on our media servers yet, so I have yet to see what the
actual performanee of netbackup and LTO3 over 10GbE is. I'm hopeful
it'll be close to the iperf results, but if it doesn't meet the goal
then we'll be looking at other options.
--
Devon Peters _______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
|
|
| Back to top |
|
|
devon
Joined: 24 Aug 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 12:24 pm Post subject: Some info on my experiences with 10GbE |
|
|
Regarding the poor performance you've seen, I'm curious how many data
streams you were using? The number of simultaneous streams seems to
have a big impact on total throughput.
Also, if you haven't done so, you should check out the Solaris Internals
website for some recommended tunables with these cards:
http://www.solarisinternals.com/wiki/index.php/Networks
I sent out the settings I'm using in a previous email, so you might be
able to take those and see if they work for you...
-devon
------------------------------
Date: Thu, 18 Oct 2007 16:30:40 +0800
From: "Mellor, Adam A." <Adam.Mellor < at > woodside.com.au>
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
To: <VERITAS-BU < at > mailman.eng.auburn.edu>
Message-ID:
<55A93A8FDF17844FBD201175FB138CC604D6F811 < at > permls07.wde.woodside.com.au>
Content-Type: text/plain; charset="us-ascii"
Also, Very Interested,
My backup servers are 8core T2000's, we will be putting the SUN nxge
10Gbit cards into them.
I have only seen poor results from the card in back to back
configuration (the network is not at 10Gbit yet). so far i have been
misserable with results in line with Mr Preston's.
More that happy (Change controll permitting) to duplicate your setup
with the 8 core boxes.
Adam.
________________________________
From: veritas-bu-bounces < at > mailman.eng.auburn.edu
[mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Curtis
Preston
Sent: Thursday, 18 October 2007 4:07 PM
To: Peters, Devon C; VERITAS-BU < at > mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
7500 MB/s! That's the most impressive numbers I've ever seen by FAR. I
may have to take back my "10 GbE is a Lie!" blog post, and I'd be happy
to do so.
Can you share things besides the T2000? For example,
what OS and patch levels are you running?
Any IP patches?
Any IP-specific patches?
What ndd settings are you using?
Is rss enabled?
"Input, I need Input!"
---
W. Curtis Preston
Backup Blog < at > www.backupcentral.com
VP Data Protection, GlassHouse Technologies
________________________________
From: veritas-bu-bounces < at > mailman.eng.auburn.edu
[mailto:veritas-bu-bounces < at > mailman.eng.auburn.edu] On Behalf Of Peters,
Devon C
Sent: Wednesday, October 17, 2007 12:12 PM
To: VERITAS-BU < at > mailman.eng.auburn.edu
Subject: [Veritas-bu] Some info on my experiences with 10GbE
Since I've seen a little bit of talk about 10GbE on here in the past I
figured I'd share some of my experiences...
I've recently been testing some of Sun's dual-port 10GbE NICs on some
small T2000's (1Ghz, 4-core). I'm only using a single port on each
card, and the servers are currently directly connected to each other
(waiting for my network team to get switches and fibre in place).
So far, I've been able to drive throughput between these two systems to
about 7500Mbit/sec using iperf. When the throughput gets this high, all
the cores/threads on the receiving T2000 become saturated and TCP
retransmits start climbing, but both systems remain quite responsive.
Since these are only 4-core T2000's, I would guess that the 6 or 8-core
T2000's (especially with 1.2Ghz or 1.4Ghz processors) should be capable
of more throughput, possibly near line speed.
The down side achieving this high of throughput is that it requires lots
of data streams. When transmitting with a single data stream, the most
throughput I've gotten is about 1500Mbit/sec. I only got up to
7500Mbit/s when using 64 data streams... Also, the biggest gains seem
to be in the jump from 1 to 8 data streams; with 8 streams I was able
to get throughput up to 6500Mbit/sec.
Our goal for 10GbE, is to be able to restore data from tape at a speed
of at least 2400Mbit/sec (300MB/sec). We have large daily backups
(3-4TB) that we would like to be able to restore (not backup) in a
reasonable amount of time. These restores are used to refresh our test
and development environments with current data. The actual backups are
done with array based snapshots (HDS ShadowCopy), which then get mounted
and backed up by a dedicated media server (6-core T2000). We're
currently getting about 650MB/sec of throughput with the backups (9
streams on 3 LTO3 tape drives - MPX=3 and it's very compressible data).
Going off my iperf results, the restoring this data using 9 streams
should get us well over 2400Mbit/sec. But - we haven't installed the
cards on our media servers yet, so I have yet to see what the actual
performanee of netbackup and LTO3 over 10GbE is. I'm hopeful it'll be
close to the iperf results, but if it doesn't meet the goal then we'll
be looking at other options.
--
Devon Peters
NOTICE: This email and any attachments are confidential.
They may contain legally privileged information or
copyright material. You must not read, copy, use or
disclose them without authorisation. If you are not an
intended recipient, please contact us at once by return
email and then delete both messages and all attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20071018/
73c7b0a6/attachment.html
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
Nick Majeran
Guest
|
| Posted: Thu Oct 18, 2007 1:21 pm Post subject: Some info on my experiences with 10GbE |
|
|
Devon, just a few more questions:
So you *are* using jumbo frames? I saw that it was enabled in ndd,
but you haven't mentioned it outright.
Also, what network switching equipment are you using for these tests?
Also, I'm curious, how is it that 4Gb/s LTO-3 drives can write
"faster" than 2 Gb/s with contrived data? It seems like it shouldn't
make a difference, since the data stream is compressed at the drive.
thanks!
-- nick
We've been pretty happy with the T2000's.
The tape library is an IBM 3584, the tape drives are IBM's 4Gb FC LTO-3
drives, there's a dedicated 4Gb HBA for each drive, and everything is
connected to 4Gb McData switches.
We used to have IBM's 2Gb FC LTO-3 drives, and with those the peak
performance was around 165MB/s per drive. These 4Gb drives peak at
around 265MB/s per drive, though with all 3 tape drives active, we see
throughput closer to 220MB/s per drive...I'm guessing we're bottlenecked
by the ports on our disk subsystem at the moment, but since performance
is more than acceptable we're not looking to tune this any further - at
least not until our LTO-4 drives are installed next month .
-devon
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
devon
Joined: 24 Aug 2007
Posts: 17
|
| Posted: Thu Oct 18, 2007 5:50 pm Post subject: Some info on my experiences with 10GbE |
|
|
Yep, I'm using jumbo frames. The performance was around 50% lower
without it. I'm not currently using any switches for 10GbE, the servers
are connected directly together.
Re 4Gb vs 2Gb tape drives - since the data is compressed at the drive,
we still need to be able to transfer the data to the drives as fast as
possible. The highest throughput we've been able to get with a single
2Gb fibre HBA is about 190MB/s (using multiple 2Gb disk-subsystem ports
zoned to a single HBA port). The highest throughput we've gotten with a
single 2Gb tape drive is 170MB/s. Since this is near the peak we can
get with 2Gb, I assume that the 2Gb interface on the tape drive is
what's limiting our throughput.
Also, we get about 4x compression of this data on the tapes (~1600MB on
an LTO3 tape). So, with 265MB/s at 4x compression, the physical write
speed of the drive is probably somewhere around 65MB/s (265/4). Since
the tape compression ratio has remained the same with both 2Gb and 4Gb
drives, I'd guess that the physical drive speeds with the 2Gb drives
were probably closer to 40MB/s (170/4)...
-devon
-----Original Message-----
From: Nick Majeran [mailto:nmajeran < at > gmail.com]
Sent: Thursday, October 18, 2007 2:18 PM
To: veritas-bu < at > mailman.eng.auburn.edu; Peters, Devon C
Subject: Re: [Veritas-bu] Some info on my experiences with 10GbE
Devon, just a few more questions:
So you *are* using jumbo frames? I saw that it was enabled in ndd,
but you haven't mentioned it outright.
Also, what network switching equipment are you using for these tests?
Also, I'm curious, how is it that 4Gb/s LTO-3 drives can write
"faster" than 2 Gb/s with contrived data? It seems like it shouldn't
make a difference, since the data stream is compressed at the drive.
thanks!
-- nick
We've been pretty happy with the T2000's.
The tape library is an IBM 3584, the tape drives are IBM's 4Gb FC LTO-3
drives, there's a dedicated 4Gb HBA for each drive, and everything is
connected to 4Gb McData switches.
We used to have IBM's 2Gb FC LTO-3 drives, and with those the peak
performance was around 165MB/s per drive. These 4Gb drives peak at
around 265MB/s per drive, though with all 3 tape drives active, we see
throughput closer to 220MB/s per drive...I'm guessing we're bottlenecked
by the ports on our disk subsystem at the moment, but since performance
is more than acceptable we're not looking to tune this any further - at
least not until our LTO-4 drives are installed next month .
-devon
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
Nick Majeran
Guest
|
| Posted: Fri Oct 19, 2007 8:03 am Post subject: Some info on my experiences with 10GbE |
|
|
Regarding the tape drives and compression -- this is the part that confuses me.
I can max-out an LTO-3 drive at native write speed at 80MB/s with no
problem using pre-compressed data (compressed Sybase dbdumps), even
with a measly 64kb block size. This is using direct NDMP with 2 Gb/s
fc IBM LTO-3 drives.
Using contrived data, i.e. large files dd'ed from /dev/zero or
hpcreatedata, I have in the past maxed out 2 Gb/s LTO-3 drives at
approximately 170 MB/s, as you claim above. However, this was using
256kb block sizes. I have read reports where 2 Gb/s LTO-3 drives can
be pushed to 220-230 MB/s using the maximum block size supported by
LTO-3 (2 MB) and contrived data.
Now, if compression is done at the drive, I would think that with a 2
Gb/s interface, it should be able to receive data at roughly 170 MB/s,
but since the drive natively spins at 80 MB/s, it would compress that
data, 4x, as you claim, to get that 240 MB/s top end. But, in my
mind, using 2 Gb/s or 4 Gb/s shouldn't make a bit of difference for a
drive that natively writes at 80 MB/s.
Does anyone else have experience with this?
Also, I've seen LTO-3 tapes in our environment marked as "FULL" by
Netbackup with close to 2 TB of data on them.
-- nick
Yep, I'm using jumbo frames. The performance was around 50% lower
without it. I'm not currently using any switches for 10GbE, the servers
are connected directly together.
Re 4Gb vs 2Gb tape drives - since the data is compressed at the drive,
we still need to be able to transfer the data to the drives as fast as
possible. The highest throughput we've been able to get with a single
2Gb fibre HBA is about 190MB/s (using multiple 2Gb disk-subsystem ports
zoned to a single HBA port). The highest throughput we've gotten with a
single 2Gb tape drive is 170MB/s. Since this is near the peak we can
get with 2Gb, I assume that the 2Gb interface on the tape drive is
what's limiting our throughput.
Also, we get about 4x compression of this data on the tapes (~1600MB on
an LTO3 tape). So, with 265MB/s at 4x compression, the physical write
speed of the drive is probably somewhere around 65MB/s (265/4). Since
the tape compression ratio has remained the same with both 2Gb and 4Gb
drives, I'd guess that the physical drive speeds with the 2Gb drives
were probably closer to 40MB/s (170/4)...
-devon
_______________________________________________
Veritas-bu maillist - Veritas-bu < at > mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu |
|
| Back to top |
|
|
|
FAQ
Search
Register
Log in to check your private messages
Log in
