Backup Central

OK, here's another oddity.
Customer upgraded from 5.5 on Win2K3, to 6.2.2 on Win2K8, 32G Ram.
Tapes are 3 IBM LTO4's in a TS3310.

Everything works great, except we appear to have a problem with BACKUP STGPOOL running very slowly, on occasion, going tape to tape.

Other tape to tape processes (reclaim, for example) work fine. BACKUP STGPOOL running from diskpool to tape works fine. But on some days, BACKUP STGPOOL running from tape to tape appears to just hang. The output from Q PROC below shows it hanging on the same "current physical file" of 122GB, for over 12 hours, until we had to cancel it.

There were no other tape processes running, nor was expiration. Over the 12 hours there were occasional small client backups, but nothing intensive. Sometimes after 3-4 hours, the process will "break free" and pick up speed again, other days it won't. Dedup not turned on. At first I suspected a bad LTO cartridge or a firmware problem, but we've updated the firmware, and this occurs on so many different cartridges that we've ruled that out. I'm stumped.

I've calculated MB/sec throughput for every combination of disk-to-tape and tape-to-tape processes and we are getting decent throughput, even on BACKUP STGPOOL, except for these occasional hangups.

Anybody seen anything like this? Otherwise I'm thinking it's time for a PMR call and a trace...

q proc

Process Process Description Status
Number
-------- -------------------- -------------------------------------------------
64 Backup Storage Pool Primary Pool ORA-LTO4POOL, Copy Pool
LTO4VAULT, Files Backed Up: 322963, Bytes Backed
Up: 2,178,940,551,897, Unreadable Files: 0,
Unreadable Bytes: 0. Current Physical File
(bytes): 122,700,012,064 Current input volume:
A00575L4. Current output volume(s): A00578L4.





Wanda Prather | Senior Technical Specialist | wprather < at > icfi.com<mailto:wprather < at > icfi.com> | www.icf.com<http://www.icf.com>
ICF International | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410.539.1135 (o)
Connect with us on social media<http://www.icfi.com/social>

I have no reason to suspect a DB2 interaction, but at the same time, with our V6 systems, our second instinct is to look in db2diag.log, just in case.

You might check your db2reorg settings, just because.


On Nov 26, 2011, at 1:23 PM, Prather, Wanda wrote:

OK, here's another oddity.
Customer upgraded from 5.5 on Win2K3, to 6.2.2 on Win2K8, 32G Ram.
Tapes are 3 IBM LTO4's in a TS3310.

Everything works great, except we appear to have a problem with BACKUP STGPOOL running very slowly, on occasion, going tape to tape.

Other tape to tape processes (reclaim, for example) work fine. BACKUP STGPOOL running from diskpool to tape works fine. But on some days, BACKUP STGPOOL running from tape to tape appears to just hang. The output from Q PROC below shows it hanging on the same "current physical file" of 122GB, for over 12 hours, until we had to cancel it.

There were no other tape processes running, nor was expiration. Over the 12 hours there were occasional small client backups, but nothing intensive. Sometimes after 3-4 hours, the process will "break free" and pick up speed again, other days it won't. Dedup not turned on. At first I suspected a bad LTO cartridge or a firmware problem, but we've updated the firmware, and this occurs on so many different cartridges that we've ruled that out. I'm stumped.

I've calculated MB/sec throughput for every combination of disk-to-tape and tape-to-tape processes and we are getting decent throughput, even on BACKUP STGPOOL, except for these occasional hangups.

Anybody seen anything like this? Otherwise I'm thinking it's time for a PMR call and a trace...

q proc

Process Process Description Status
Number
-------- -------------------- -------------------------------------------------
64 Backup Storage Pool Primary Pool ORA-LTO4POOL, Copy Pool
LTO4VAULT, Files Backed Up: 322963, Bytes Backed
Up: 2,178,940,551,897, Unreadable Files: 0,
Unreadable Bytes: 0. Current Physical File
(bytes): 122,700,012,064 Current input volume:
A00575L4. Current output volume(s): A00578L4.

Hi Wanda,

We run v6.2.3 and have had an open PMR for months now about performance issues, mostly concerning automatic reorganization, but that lead to our questioning whether it was causing other issues. As of now, we are being told by TSM developers that we need to adjust the DB2 buffer pools to give us better reorganization performance, which in turn, they say, should also give us better performance with our other admin tasks, but we are being told by the DB2 developers that they recommend that we stay with the Automatic settings for the buffer pools, or take our chances with changing them ourselves. Neither team seems to be able to help us with what our optimal settings should be.

As you can imagine we are not in a happy place at the moment. I would definitely open a PMR. There are known issues about administrative task performance.

Best regards,
Pam

Pam Pagnotta
Sr. Systems Engineer
Energy Enterprise Solutions (EES), LLC
Supporting IM-621.1, Enterprise Service Center East
Contractor to the U.S. Department of Energy
Office: 301-903-5508
Email: pam.pagnotta < at > hq.doe.gov
Location: USA (EST/EDT)



----------------------------------------------------------
Date: Sat, 26 Nov 2011 19:23:53 +0000
From: "Prather, Wanda" <wPrather < at > ICFI.COM>
Subject: 6.2.2 Backup Stgpool Performance issue?

OK, here's another oddity.
Customer upgraded from 5.5 on Win2K3, to 6.2.2 on Win2K8, 32G Ram.
Tapes are 3 IBM LTO4's in a TS3310.

Everything works great, except we appear to have a problem with BACKUP STGP= OOL running very slowly, on occasion, going tape to tape.

Other tape to tape processes (reclaim, for example) work fine. BACKUP STGP= OOL running from diskpool to tape works fine. But on some days, BACKUP STG= POOL running from tape to tape appears to just hang. The output from Q PRO= C below shows it hanging on the same "current physical file" of 122GB, for = over 12 hours, until we had to cancel it.

There were no other tape processes running, nor was expiration. Over the 1=
2 hours there were occasional small client backups, but nothing intensive. = Sometimes after 3-4 hours, the process will "break free" and pick up speed= again, other days it won't. Dedup not turned on. At first I suspected a = bad LTO cartridge or a firmware problem, but we've updated the firmware, an= d this occurs on so many different cartridges that we've ruled that out. I= 'm stumped.

I've calculated MB/sec throughput for every combination of disk-to-tape and= tape-to-tape processes and we are getting decent throughput, even on BACKU= P STGPOOL, except for these occasional hangups.

Anybody seen anything like this? Otherwise I'm thinking it's time for a PM= R call and a trace...

q proc

Process Process Description Status
Number
-------- -------------------- -------------------------------------=
------------
64 Backup Storage Pool Primary Pool ORA-LTO4POOL, Copy Pool
LTO4VAULT, Files Backed Up: 322963, = Bytes Backed
Up: 2,178,940,551,897, Unreadable Fi=
les: 0,
Unreadable Bytes: 0. Current Physical= File
(bytes): 122,700,012,064 Current inp= ut volume:
A00575L4. Current output volume(s): = A00578L4.





Wanda Prather | Senior Technical Specialist | wprather < at > icfi.com<mailto:w= prather < at > icfi.com> | www.icf.com<http://www.icf.com> ICF International | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410=
.539.1135 (o)
Connect with us on social media<http://www.icfi.com/social>

I wonder if the performance issue you've seen is related to my discovery that client backup has slowed to an unacceptable level?

Network throughput rates of 2,500 KB/sec and higher are now only 400 KB/sec or slower.

I'm planning to open a PMR this week.

------------------------------------------------
Harold Vandeventer

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L < at > VM.MARIST.EDU] On Behalf Of Pagnotta, Pam (CONTR)
Sent: Monday, November 28, 2011 8:16 AM
To: ADSM-L < at > VM.MARIST.EDU
Subject: [ADSM-L] 6.2.2 Backup Stgpool Performance issue?

Hi Wanda,

We run v6.2.3 and have had an open PMR for months now about performance issues, mostly concerning automatic reorganization, but that lead to our questioning whether it was causing other issues. As of now, we are being told by TSM developers that we need to adjust the DB2 buffer pools to give us better reorganization performance, which in turn, they say, should also give us better performance with our other admin tasks, but we are being told by the DB2 developers that they recommend that we stay with the Automatic settings for the buffer pools, or take our chances with changing them ourselves. Neither team seems to be able to help us with what our optimal settings should be.

As you can imagine we are not in a happy place at the moment. I would definitely open a PMR. There are known issues about administrative task performance.

Best regards,
Pam

Pam Pagnotta
Sr. Systems Engineer
Energy Enterprise Solutions (EES), LLC
Supporting IM-621.1, Enterprise Service Center East
Contractor to the U.S. Department of Energy
Office: 301-903-5508
Email: pam.pagnotta < at > hq.doe.gov
Location: USA (EST/EDT)



----------------------------------------------------------
Date: Sat, 26 Nov 2011 19:23:53 +0000
From: "Prather, Wanda" <wPrather < at > ICFI.COM>
Subject: 6.2.2 Backup Stgpool Performance issue?

OK, here's another oddity.
Customer upgraded from 5.5 on Win2K3, to 6.2.2 on Win2K8, 32G Ram.
Tapes are 3 IBM LTO4's in a TS3310.

Everything works great, except we appear to have a problem with BACKUP STGP= OOL running very slowly, on occasion, going tape to tape.

Other tape to tape processes (reclaim, for example) work fine. BACKUP STGP= OOL running from diskpool to tape works fine. But on some days, BACKUP STG= POOL running from tape to tape appears to just hang. The output from Q PRO= C below shows it hanging on the same "current physical file" of 122GB, for = over 12 hours, until we had to cancel it.

There were no other tape processes running, nor was expiration. Over the 1=
2 hours there were occasional small client backups, but nothing intensive. = Sometimes after 3-4 hours, the process will "break free" and pick up speed= again, other days it won't. Dedup not turned on. At first I suspected a = bad LTO cartridge or a firmware problem, but we've updated the firmware, an= d this occurs on so many different cartridges that we've ruled that out. I= 'm stumped.

I've calculated MB/sec throughput for every combination of disk-to-tape and= tape-to-tape processes and we are getting decent throughput, even on BACKU= P STGPOOL, except for these occasional hangups.

Anybody seen anything like this? Otherwise I'm thinking it's time for a PM= R call and a trace...

q proc

Process Process Description Status
Number
-------- -------------------- -------------------------------------=
------------
64 Backup Storage Pool Primary Pool ORA-LTO4POOL, Copy Pool
LTO4VAULT, Files Backed Up: 322963, = Bytes Backed
Up: 2,178,940,551,897, Unreadable Fi=
les: 0,
Unreadable Bytes: 0. Current Physical= File
(bytes): 122,700,012,064 Current inp= ut volume:
A00575L4. Current output volume(s): = A00578L4.





Wanda Prather | Senior Technical Specialist | wprather < at > icfi.com<mailto:w= prather < at > icfi.com> | www.icf.com<http://www.icf.com> ICF International | 401 E. Pratt St, Suite 2214, Baltimore, MD 21202 | 410=
.539.1135 (o)
Connect with us on social media<http://www.icfi.com/social>