Backup Central

We have the following environment in brief: A sun V440 with 4 CPUs, and
8GB RAM. To this server, a SpectraLogic T950 is attached via a fibre
channel loop. We have a dual port Sun HBA adapter (SG-XPCI2FC-JF2) that
uses both ports to reach different drives in the library. It is also
running Sun Trunking Software v1.3 to connect to the network, since both
ce interfaces are bundled as a port-channel. Both interfaces are running
1000mbit/fdx. The load-balance algorithm used for outbound traffic is
ip-source-dest pair hashing. We're running Solaris 9, with latest patches
for OS, HBA etc. Some tweaks have been applied to /etc/system, such as
maxphys, number of file descriptors, maxusers etc. The HBA is on its own
PCI bus in the correct PCI slot type.

The problem is that we can't get maximum performance out of all 6 SAIT-1
drives at once. When the server transits about 900mbit of data, the CPU
load has reached a point where there are no more cycles available. At that
time, the user space processes account for approx 15% and the kernel for
approx 80%.

I'm wondering why the kernel consumes so much CPU, considering the
relatively low throughput? What part could be causing this? Is it the LUS
driver, or the HBA driver? Or something else? Can one find out?

I was thinking that maybe the HBA doesn't have any good CPU offloading
functions for handling I/O, but that's just a theory. Is there any other
way one can find out which part of the system causes such a high CPU load?
Its not user space daemons, so "top" isn't sufficient. ;)

And is it possible that it could indeed be the HBA? I mean, there is often
a TCP offloading engine on better NIC cards, and while looking at
different HBA's, it seems like those have different offloading mechanisms
as well. I've been looking mostly at Emulex cards, and they seem to differ
quite a bit when it comes to architecture.

For instance, if I have a 66MHz PCI bus, will a LP11002 still be faster
than a LP9002DC? And yes, I include lower system CPU utilization per
megabit of throughput in my definition of "faster".

(http://www.emulex.com/products/fc/index.html)

Would be good to hear other people's experience from this..

//Oscar

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

Do you happen to use cisco switches? If so do you know what they are
set to, and have you made sure your getting connections through both
sides of the trunk? I'm in the middle of configuring a V880 with
trunked NICs, but can't get data down both paths.

Later,

Joel

-----Original Message-----
From: Legato NetWorker discussion [mailto:NETWORKER < at > LISTMAIL.TEMPLE.EDU]
On Behalf Of Oscar Olsson
Sent: Monday, January 03, 2005 11:41 AM
To: NETWORKER < at > LISTMAIL.TEMPLE.EDU
Subject: [Networker] General sun HW performance question

We have the following environment in brief: A sun V440 with 4 CPUs, and
8GB RAM. To this server, a SpectraLogic T950 is attached via a fibre
channel loop. We have a dual port Sun HBA adapter (SG-XPCI2FC-JF2) that
uses both ports to reach different drives in the library. It is also
running Sun Trunking Software v1.3 to connect to the network, since both
ce interfaces are bundled as a port-channel. Both interfaces are running
1000mbit/fdx. The load-balance algorithm used for outbound traffic is
ip-source-dest pair hashing. We're running Solaris 9, with latest
patches
for OS, HBA etc. Some tweaks have been applied to /etc/system, such as
maxphys, number of file descriptors, maxusers etc. The HBA is on its own
PCI bus in the correct PCI slot type.

The problem is that we can't get maximum performance out of all 6 SAIT-1
drives at once. When the server transits about 900mbit of data, the CPU
load has reached a point where there are no more cycles available. At
that
time, the user space processes account for approx 15% and the kernel for
approx 80%.

I'm wondering why the kernel consumes so much CPU, considering the
relatively low throughput? What part could be causing this? Is it the
LUS
driver, or the HBA driver? Or something else? Can one find out?

I was thinking that maybe the HBA doesn't have any good CPU offloading
functions for handling I/O, but that's just a theory. Is there any other
way one can find out which part of the system causes such a high CPU
load?
Its not user space daemons, so "top" isn't sufficient. ;)

And is it possible that it could indeed be the HBA? I mean, there is
often
a TCP offloading engine on better NIC cards, and while looking at
different HBA's, it seems like those have different offloading
mechanisms
as well. I've been looking mostly at Emulex cards, and they seem to
differ
quite a bit when it comes to architecture.

For instance, if I have a 66MHz PCI bus, will a LP11002 still be faster
than a LP9002DC? And yes, I include lower system CPU utilization per
megabit of throughput in my definition of "faster".

(http://www.emulex.com/products/fc/index.html)

Would be good to hear other people's experience from this..

//Oscar

Note: To sign off this list, send a "signoff networker" command via
email
should be sent to stan < at > temple.edu
=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D=
*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu
=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D

On Mon, 3 Jan 2005, Joel Fisher wrote:

JF> Do you happen to use cisco switches? If so do you know what they are
JF> set to, and have you made sure your getting connections through both
JF> sides of the trunk? I'm in the middle of configuring a V880 with
JF> trunked NICs, but can't get data down both paths.

Yup. I MRTG graph both physical ports, and both inbound and outbound
traffic seems to flow over both interfaces. Have you configured the sun
trunking software correctly?

This is how my config looks:

On the cisco switch (relevant parts):

interface Port-channel2
description Port-channel to britt.qbranch.se
no ip address
no logging event link-status
switchport
switchport access vlan 4
switchport mode access

interface GigabitEthernet3/4
description Interface 1 of 2 for port channel 2 to britt.qbranch.se
no ip address
no logging event link-status
flowcontrol receive desired
switchport
switchport access vlan 4
switchport mode access
no cdp enable
channel-group 2 mode on
!
interface GigabitEthernet3/5
description Interface 2 of 2 for port channel 2 to britt.qbranch.se
no ip address
no logging event link-status
flowcontrol receive desired
switchport
switchport access vlan 4
switchport mode access
no cdp enable
channel-group 2 mode on
!

The cisco switch defaults to ip-source-dest address hashing for path
selection, so that's why it isn't mentioned above. Its a catalyst 6509
with SUP-1A running native IOS. Note that the port must be in mode "on"
for port channels. Suns software does not support auto negotiation of port
channels.

On the backup server, it looks like this:

In /etc/opt/SUNWconn/trunking/bin/nettr.sh:

# Add Sun Trunking Configuration entries below.
nettr -setup 0 device=ce members=0,1 policy=3

Then configure ce0 just as you would if it were a single interface. Note
that ce1 must not be plumbed.

Hope this helps. :)

//Oscar

Ps. Does anyone have an answer to MY question as well? :)

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

The 440 has 2 internal gigabit adapters (ce's).

You might be hitting a Sun design issue? To drive one ce to 900mbit/s their
blueprint says one needs several UltraSparcIII CPUs:

http://www.sun.com/blueprints/0203/817-1657.pdf

See this article for some tuning that is possible;
http://www.sun.com/blueprints/0404/817-6925.pdf

Solaris 9 was suppose to have very good TCP/IP CPU utilization though?


Robert Maiello
Pioneer Data Systems


On Mon, 3 Jan 2005 17:40:34 +0100, Oscar Olsson <spam1 < at > QBRANCH.SE> wrote:

We have the following environment in brief: A sun V440 with 4 CPUs, and
8GB RAM. To this server, a SpectraLogic T950 is attached via a fibre
channel loop. We have a dual port Sun HBA adapter (SG-XPCI2FC-JF2) that
uses both ports to reach different drives in the library. It is also
running Sun Trunking Software v1.3 to connect to the network, since both
ce interfaces are bundled as a port-channel. Both interfaces are running
1000mbit/fdx. The load-balance algorithm used for outbound traffic is
ip-source-dest pair hashing. We're running Solaris 9, with latest patches
for OS, HBA etc. Some tweaks have been applied to /etc/system, such as
maxphys, number of file descriptors, maxusers etc. The HBA is on its own
PCI bus in the correct PCI slot type.

The problem is that we can't get maximum performance out of all 6 SAIT-1
drives at once. When the server transits about 900mbit of data, the CPU
load has reached a point where there are no more cycles available. At that
time, the user space processes account for approx 15% and the kernel for
approx 80%.

I'm wondering why the kernel consumes so much CPU, considering the
relatively low throughput? What part could be causing this? Is it the LUS
driver, or the HBA driver? Or something else? Can one find out?

I was thinking that maybe the HBA doesn't have any good CPU offloading
functions for handling I/O, but that's just a theory. Is there any other
way one can find out which part of the system causes such a high CPU load?
Its not user space daemons, so "top" isn't sufficient. ;)

And is it possible that it could indeed be the HBA? I mean, there is often
a TCP offloading engine on better NIC cards, and while looking at
different HBA's, it seems like those have different offloading mechanisms
as well. I've been looking mostly at Emulex cards, and they seem to differ
quite a bit when it comes to architecture.

For instance, if I have a 66MHz PCI bus, will a LP11002 still be faster
than a LP9002DC? And yes, I include lower system CPU utilization per
megabit of throughput in my definition of "faster".

(http://www.emulex.com/products/fc/index.html)

Would be good to hear other people's experience from this..

//Oscar

--
Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

On Tue, 4 Jan 2005, Robert Maiello wrote:

RM> The 440 has 2 internal gigabit adapters (ce's).
RM>
RM> You might be hitting a Sun design issue? To drive one ce to 900mbit/s their
RM> blueprint says one needs several UltraSparcIII CPUs:
RM>
RM> http://www.sun.com/blueprints/0203/817-1657.pdf

Yep, but that's for the ge adapter, and not for the ce one, and for TCP in
general.

RM> See this article for some tuning that is possible;
RM> http://www.sun.com/blueprints/0404/817-6925.pdf
RM>
RM> Solaris 9 was suppose to have very good TCP/IP CPU utilization though?

I read this document, and checked if I had any problems that are mentioned
in this document. I couldn't find any signs of PCI bus congestion or any
signs of other issues that are mentioned in this document.

I think the core essence of my problem is to find out what part of the
system causes such a high load. Is it the NIC or the FC adapter? And if it
is either one, can it be replaced with a better adapter that has a better
driver and/or hardware that creates less load on the system?

Since its a V440 server with 4 CPUs, I can't add more CPU power. But
either way, I think its pathetic if you require more CPU power than that
just to drive a gig or two of I/O.

//Oscar

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

I'd recommend you contact Legato support. They have some utilities that
can help to isolate the bottleneck. =20

-Teresa

-----Original Message-----
From: Legato NetWorker discussion [mailto:NETWORKER < at > LISTMAIL.TEMPLE.EDU]
On Behalf Of Oscar Olsson
Sent: Wednesday, January 05, 2005 3:36 AM
To: NETWORKER < at > LISTMAIL.TEMPLE.EDU
Subject: Re: [Networker] General sun HW performance question

.
.
.

I think the core essence of my problem is to find out what part of the
system causes such a high load. Is it the NIC or the FC adapter? And if
it
is either one, can it be replaced with a better adapter that has a
better
driver and/or hardware that creates less load on the system?

.
.
.

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu
=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D*=3D

On Wed, 5 Jan 2005 09:35:57 +0100, Oscar Olsson <spam1 < at > QBRANCH.SE> wrote:

On Tue, 4 Jan 2005, Robert Maiello wrote:

RM> The 440 has 2 internal gigabit adapters (ce's).
<SNIP>
RM> Solaris 9 was suppose to have very good TCP/IP CPU utilization though?

I read this document, and checked if I had any problems that are mentioned
in this document. I couldn't find any signs of PCI bus congestion or any
signs of other issues that are mentioned in this document.

I think the core essence of my problem is to find out what part of the
system causes such a high load. Is it the NIC or the FC adapter? And if it
is either one, can it be replaced with a better adapter that has a better
driver and/or hardware that creates less load on the system?

Since its a V440 server with 4 CPUs, I can't add more CPU power. But
either way, I think its pathetic if you require more CPU power than that
just to drive a gig or two of I/O.

//Oscar

Use bigasm on the backup server to hammer your tape drives, then use it
from a fast client to hammer the network this should help identify which
is a bottle neck.

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

On Wed, 5 Jan 2005, Howard Martin wrote:

HM> On Wed, 5 Jan 2005 09:35:57 +0100, Oscar Olsson <spam1 < at > QBRANCH.SE> wrote:
HM> >On Tue, 4 Jan 2005, Robert Maiello wrote:
HM> >RM> The 440 has 2 internal gigabit adapters (ce's).
HM> <SNIP>
HM> >RM> Solaris 9 was suppose to have very good TCP/IP CPU utilization though?
HM> >
HM> >I read this document, and checked if I had any problems that are mentioned
HM> >in this document. I couldn't find any signs of PCI bus congestion or any
HM> >signs of other issues that are mentioned in this document.
HM> >
HM> >I think the core essence of my problem is to find out what part of the
HM> >system causes such a high load. Is it the NIC or the FC adapter? And if it
HM> >is either one, can it be replaced with a better adapter that has a better
HM> >driver and/or hardware that creates less load on the system?
HM> >
HM> >Since its a V440 server with 4 CPUs, I can't add more CPU power. But
HM> >either way, I think its pathetic if you require more CPU power than that
HM> >just to drive a gig or two of I/O.
HM> >
HM> >//Oscar
HM> >
HM> Use bigasm on the backup server to hammer your tape drives, then use it
HM> from a fast client to hammer the network this should help identify which
HM> is a bottle neck.

Interesting!

I created five directories in the root of the backup server, and then
creating a local bigadm directive for a file in each directive, making the
networker server back up 50GB in each directory. Then I created five
instances of the backup server client definition, each backing up one
directory to a different pool, thus making each saveset go to a different
drive. Then I ran these five new definitions simultaneously in five
different groups, making them write data to five different drives
simultaneously.

I now see a throughput of ~1,5gigabit, which is close to the drives native
capacity (S-AIT1), but since I'm not sure of the contents of the actual
data being written, I'm assuming that compression isn't effective on this
type of data.

Now top shows this more or less continously:

last pid: 8004; load averages: 0.39, 0.40, 0.38 12:52:09
78 processes: 77 sleeping, 1 on cpu
CPU states: 19.3% idle, 2.5% user, 4.9% kernel, 73.3% iowait, 0.0% swap
Memory: 8192M real, 7203M free, 224M swap in use, 14G swap free

Which looks much better, since this indicates that the drives are the
bottlenecks in this case.


Now to the next question. Why does the network I/O create such a high CPU
utilization? Is it because the ce adapters lack hardware acceleration
features, or is it because the Solaris 9 IP-stack isn't very fast?

I have read the previous suggested documents about IP stack and OS tuning,
and I have applied some changes that I see fit, according to the various
sources. This has boosted performance with say ~5-10%, but I'm looking for
more like 50-100%, since that's what the drives should be able to handle
on average with live data comming from the clients.

So, is this high load by design, or are there better network adapters,
that can offload the CPU better than the ce ones, or is perhaps Solaris 10
an alternative, since SUn boasts how they have improved the IP stack? I
really don't know where to look next. I can't add more CPUs, since there
are already 4 CPUs in my V440 system. And upgrading the CPUs will probably
be too expensive, compared to the expected performance boost.

//Oscar

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

Oscar Olsson wrote:

On Wed, 5 Jan 2005, Howard Martin wrote:



Now to the next question. Why does the network I/O create such a high CPU
utilization? Is it because the ce adapters lack hardware acceleration
features, or is it because the Solaris 9 IP-stack isn't very fast?




You have mentioned you run Sun Trunking. IIRC that software can only run
on old Sun GEM gigabit cards (driver ge).
Those cards produce a lot of interrupts so they are not very efficient
in terms of CPU consumption.
You'd better of with newer chipsets, but not sure if Sun Trunking now
supports them.
In any case I know there are other 3d party alternatives to Sun Trunking
that work with many (if not all) NICs.

I have read the previous suggested documents about IP stack and OS tuning,
and I have applied some changes that I see fit, according to the various
sources. This has boosted performance with say ~5-10%, but I'm looking for
more like 50-100%, since that's what the drives should be able to handle
on average with live data comming from the clients.

So, is this high load by design, or are there better network adapters,
that can offload the CPU better than the ce ones, or is perhaps Solaris 10
an alternative, since SUn boasts how they have improved the IP stack? I
really don't know where to look next. I can't add more CPUs, since there
are already 4 CPUs in my V440 system. And upgrading the CPUs will probably
be too expensive, compared to the expected performance boost.

//Oscar

--
Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu



Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

Now to the next question. Why does the network I/O create such a high CPU
utilization? Is it because the ce adapters lack hardware acceleration
features, or is it because the Solaris 9 IP-stack isn't very fast?




You have mentioned you run Sun Trunking. IIRC that software can only run
on old Sun GEM gigabit cards (driver ge).

More recent versions of SunTrunking have supported ce devices for some
time now.

Those cards produce a lot of interrupts so they are not very efficient
in terms of CPU consumption.
You'd better of with newer chipsets, but not sure if Sun Trunking now
supports them.
In any case I know there are other 3d party alternatives to Sun Trunking
that work with many (if not all) NICs.

Do you have any examples? I'd love to know about them.

IPMP can do some of what SunTrunking does, but not all of it. It's
especially poor for a backup server which would like to aggregate lots
of incoming traffic from a small number of clients.

Darren Dunham ddunham < at > taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

On Thu, 6 Jan 2005, Yura Pismerov wrote:

YP> You have mentioned you run Sun Trunking. IIRC that software can only run
YP> on old Sun GEM gigabit cards (driver ge).
YP> Those cards produce a lot of interrupts so they are not very efficient
YP> in terms of CPU consumption.
YP> You'd better of with newer chipsets, but not sure if Sun Trunking now
YP> supports them.
YP> In any case I know there are other 3d party alternatives to Sun Trunking
YP> that work with many (if not all) NICs.

Well, I have two ce adapters, and according to the docs, sun trunking
supports them, and either way it works. Traffic flows through both
interfaces, and that is true for both incoming and outbound packets.

It would be great if you would send some URLs to these other products as
well, or at least mention their names, so I can google for them. :)

//Oscar

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

On Thu, 6 Jan 2005, Darren Dunham wrote:

DD> IPMP can do some of what SunTrunking does, but not all of it. It's
DD> especially poor for a backup server which would like to aggregate lots
DD> of incoming traffic from a small number of clients.

Since its not etherchannel and the NICs require a unique MAC address, I
don't see how the switch could load balance the incoming traffic. I think
that's why I dismissed that feature when setting up the server to begin
with, but I can't remember for sure. Anyone who has more
information/insight could perhaps enlighen us. :)

//Oscar

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

On Thu, 6 Jan 2005, Darren Dunham wrote:

DD> IPMP can do some of what SunTrunking does, but not all of it. It's
DD> especially poor for a backup server which would like to aggregate lots
DD> of incoming traffic from a small number of clients.

Since its not etherchannel and the NICs require a unique MAC address, I
don't see how the switch could load balance the incoming traffic.

Thats exactly why it's poor for that purpose.

If you have large numbers of clients, then you can balance them across
multiple IPs. But it's not as easy as having it done automatically for
you.

Darren Dunham ddunham < at > taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu

It is good to know about se support !
I think I may now review my plans for SunTrunkng. I did not consider
it because of the limited choice of the NICs it used to support.
Thanks for the info !
As for the mentioned 3d party solutions, I sent you another mail.
Even though I don't affiliate with that company I hate to "spam" mailing
lists...


Darren Dunham wrote:

Now to the next question. Why does the network I/O create such a high CPU
utilization? Is it because the ce adapters lack hardware acceleration
features, or is it because the Solaris 9 IP-stack isn't very fast?





You have mentioned you run Sun Trunking. IIRC that software can only run
on old Sun GEM gigabit cards (driver ge).



More recent versions of SunTrunking have supported ce devices for some
time now.



Those cards produce a lot of interrupts so they are not very efficient
in terms of CPU consumption.
You'd better of with newer chipsets, but not sure if Sun Trunking now
supports them.
In any case I know there are other 3d party alternatives to Sun Trunking
that work with many (if not all) NICs.



Do you have any examples? I'd love to know about them.

IPMP can do some of what SunTrunking does, but not all of it. It's
especially poor for a backup server which would like to aggregate lots
of incoming traffic from a small number of clients.

--
Darren Dunham ddunham < at > taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

--
Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu



Note: To sign off this list, send a "signoff networker" command via email
should be sent to stan < at > temple.edu