Backup Central

We have a devtype=file stgpool that is on NAS disk, accessed via NFS,
and we're getting very slow performance with TSM reading or writing it.
In a test, the Unix cp command moved data about 5 times faster than TSM
Migration.

I have been adjusting the number of migration processes up and down,
while watching network data flow numbers with topas, trying to get
clues. There comes a point, that processor wait time goes over 90%, and
then it hits a wall. The maximum seems to be about 10Mbytes/sec on a
point-to-point network consisting of three trunked GigE connections. A
single Unix cp command could write about 48Mbytes/sec, to the same NFS
filesystem, on the same NFS server, across the same network.

Anybody else faced this kind of issue?

Roger Deschner University of Illinois at Chicago rogerd < at > uic.edu
Academic Computing & Communications Center

Yep, had the same problem with iSCSI volumes. Try turning off
directio with the following undocumented dsmserv.opt option.

directio no

Gary Bowers
Storage Architect
Itrus Technologies

On Mar 1, 2010, at 10:24 PM, Roger Deschner wrote:

We have a devtype=file stgpool that is on NAS disk, accessed via NFS,
and we're getting very slow performance with TSM reading or writing
it.
In a test, the Unix cp command moved data about 5 times faster than
TSM
Migration.

I have been adjusting the number of migration processes up and down,
while watching network data flow numbers with topas, trying to get
clues. There comes a point, that processor wait time goes over 90%,
and
then it hits a wall. The maximum seems to be about 10Mbytes/sec on a
point-to-point network consisting of three trunked GigE connections. A
single Unix cp command could write about 48Mbytes/sec, to the same NFS
filesystem, on the same NFS server, across the same network.

Anybody else faced this kind of issue?

Roger Deschner University of Illinois at Chicago rogerd < at > uic.edu
Academic Computing & Communications Center

On Mar 1, 2010, at 10:24 PM, Roger Deschner wrote:
then it hits a wall. The maximum seems to be about 10Mbytes/sec on a
point-to-point network consisting of three trunked GigE connections.

I'm interested in how you have the NAS connected to your TSM server.

By "point-to-point" do you mean direct cross over cables directly
attached between the NAS and your TSM server - ie: not going
through an ethernet switches?

With single source/destinitation IP/MAC addresses, what
distribution policy do you have set on the trunk to get
load balancing? Is it set for round-robin?

Thanks

Rick

-----------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If
the reader of this message is not the intended recipient or an
agent responsible for delivering it to the intended recipient, you
are hereby notified that you have received this document in error
and that any review, dissemination, distribution, or copying of
this message is strictly prohibited. If you have received this
communication in error, please notify us immediately, and delete
the original message.

Well THAT was dramatic! Thanks, Gary.

With DIRECTIO NO set, the write speed to NFS got a LOT faster. However,
the system also started to page memory very heavily. Watching it with
topas, migration would write to NFS nicely for about 15 seconds, and
then it would stop for 5 seconds while AIX thrashed. This cycle repeats
every 20 seconds or so. Before seting DIRECTIO NO, this system didn't
page at all. It's still a large net improvement, even with the memory
thrashing.

I'm now adjusting the number of migration processes to see what number
produces the greatest total throughput, with only a tolerable amount of
thrashing. I'm also looking into AIX memory management settings.

I'm also going to watch database I/O performance closely. I've heard it
can be slowed by DIRECTIO NO.

Roger Deschner University of Illinois at Chicago rogerd < at > uic.edu
Academic Computing & Communications Center


On Tue, 2 Mar 2010, Gary Bowers wrote:

Yep, had the same problem with iSCSI volumes. Try turning off
directio with the following undocumented dsmserv.opt option.

directio no

Gary Bowers
Storage Architect
Itrus Technologies

On Mar 1, 2010, at 10:24 PM, Roger Deschner wrote:

We have a devtype=file stgpool that is on NAS disk, accessed via NFS,
and we're getting very slow performance with TSM reading or writing
it.
In a test, the Unix cp command moved data about 5 times faster than
TSM
Migration.

I have been adjusting the number of migration processes up and down,
while watching network data flow numbers with topas, trying to get
clues. There comes a point, that processor wait time goes over 90%,
and
then it hits a wall. The maximum seems to be about 10Mbytes/sec on a
point-to-point network consisting of three trunked GigE connections. A
single Unix cp command could write about 48Mbytes/sec, to the same NFS
filesystem, on the same NFS server, across the same network.

Anybody else faced this kind of issue?

Roger Deschner University of Illinois at Chicago rogerd < at > uic.edu
Academic Computing & Communications Center

On 4 mrt 2010, at 05:23, Roger Deschner wrote:

Well THAT was dramatic! Thanks, Gary.

With DIRECTIO NO set, the write speed to NFS got a LOT faster. However,
the system also started to page memory very heavily. Watching it with
topas, migration would write to NFS nicely for about 15 seconds, and
then it would stop for 5 seconds while AIX thrashed. This cycle repeats
every 20 seconds or so. Before seting DIRECTIO NO, this system didn't
page at all. It's still a large net improvement, even with the memory
thrashing.


Be aware, 'normal' filesystem I/O is handled trough the kernel paging mechanism, this is normal, expected behavior. This is actually the major difference between direct i/o and normal i/o.

What you need to check is that the amount of memory set aside for filesystem cache is not so big that AIX is paging TSM bufferpool or application memory. There are some tuning variables you can set, I forgot what they are called. Using nmon or topas you can check that you still have enough free memory.


I'm now adjusting the number of migration processes to see what number
produces the greatest total throughput, with only a tolerable amount of
thrashing. I'm also looking into AIX memory management settings.

I'm also going to watch database I/O performance closely. I've heard it
can be slowed by DIRECTIO NO.





--

Met vriendelijke groeten/Kind regards,

Remco Post