Welcome! » Log In » Create A New Profile

ISILON storage/FILE DEVCLASS performance issues

Posted by Zoltan Forray 
Zoltan Forray
ISILON storage/FILE DEVCLASS performance issues
May 11, 2018 08:59AM
Folks,

ISP 7.1.7.300 on RHEL 6 10G connectivity

We need some guidance on trying to figure out why ISP/TSM write perform to
ISILON storage (via FILE DEVCLASS) is so horrible.

We recently attached 200TB of ISILON storage to this server so we could
empty the 36TB of onboard disk drives to move this server to new hardware.

However, per my OS and SAN folks, we are only seeing 1Gbs level of data
movement from the ISP server. Doing a regular file copy to this same
storage peaks at 10Gbs speeds.

So what, if anything, are we doing wrong when it comes to configuring the
storage for ISP to use? Are there some secret controls/settings/options to
tell it to use the storage at max-speeds?

We tried changing the Est/Max capacity thinking larger files would reduce
the overhead of allocating new pieces constantly. Changed the Mount Limit
to a bigger number. Nothing has helped.

Only thing uses the storage right now is migrations from the original disk
stgpool.



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Harris, Steven
Re: ISILON storage/FILE DEVCLASS performance issues
May 13, 2018 05:59PM
Zoltan

I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual 10Gb links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX only supports NFS3, and as others have pointed out in this forum recently, the stack does not have a good reputation.

I'm finding that the heavy NFS load has other knock on effects, e.g. TSMManager keeps reporting the instance offline when it's very busy as it gets a network error on some of its regular queries, but these work ok when load is light. Also getting a lot of Severed/reconnected sessions. CPU/IO/Paging are not a problem.

Cheers

Steve

Steven Harris
TSM Admin/Consultant
Canberra Australia

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Zoltan Forray
Sent: Saturday, 12 May 2018 1:39 AM
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues

Folks,

ISP 7.1.7.300 on RHEL 6 10G connectivity

We need some guidance on trying to figure out why ISP/TSM write perform to ISILON storage (via FILE DEVCLASS) is so horrible.

We recently attached 200TB of ISILON storage to this server so we could empty the 36TB of onboard disk drives to move this server to new hardware.

However, per my OS and SAN folks, we are only seeing 1Gbs level of data movement from the ISP server. Doing a regular file copy to this same storage peaks at 10Gbs speeds.

So what, if anything, are we doing wrong when it comes to configuring the storage for ISP to use? Are there some secret controls/settings/options to tell it to use the storage at max-speeds?

We tried changing the Est/Max capacity thinking larger files would reduce the overhead of allocating new pieces constantly. Changed the Mount Limit to a bigger number. Nothing has helped.

Only thing uses the storage right now is migrations from the original disk stgpool.



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/


This message and any attachment is confidential and may be privileged or otherwise protected from disclosure. You should immediately delete the message if you are not the intended recipient. If you have received this email by mistake please delete it from your system; you should not copy the message or disclose its content to anyone.

This electronic communication may contain general financial product advice but should not be relied upon or construed as a recommendation of any financial product. The information has been prepared without taking into account your objectives, financial situation or needs. You should consider the Product Disclosure Statement relating to the financial product and consult your financial adviser before making a decision about whether to acquire, hold or dispose of a financial product.

For further details on the financial product please go to http://www.bt.com.au

Past performance is not a reliable indicator of future performance.
This message was imported via the External PhorumMail Module
AIX does supported NFSv4. I'd love to get the time to try it out, but DataDomains only support NVSv4 with the newest release and we are several levels back.
https://www.redbooks.ibm.com/redbooks/pdfs/sg246657.pdf

When our admins found that multiple/concurrent mount points helped for Oracle/RMAN backups, I converted out DataDomain mounts from single mounts to multiple mounts. I then changed the file pool devclass to use multiple directories (one per new mount). I pulled the original directory out of the file pool definition, marked all it's volumes R/O, and movedata all vols. TSM put the vols across the multiple mount points. Lets just say this took quite some time! The scary things was convincing myself (via testing) that I could remove a directory from the devclass and still access those volumes.


original single mount:
ddxxxx-10g:/data/col1/tsm1 1274748715008 323228344320 75% 103091 1% /DD/tsm1
file pool was at /DD/tsm1/backup-pri-dd

changed to these mounts
ddxxxx-10g:/data/col1/tsm1 1274748715008 323228344320 75% 103091 1% /DD/tsm1
ddxxxx-10g:/data/col1/tsm1/backup-pri-dd-1 1274748715008 323228344320 75% 103091 1% /DD/backup-pri-dd-1
ddxxxx-10g:/data/col1/tsm1/backup-pri-dd-2 1274748715008 323228344320 75% 103091 1% /DD/backup-pri-dd-2
ddxxxx-10g:/data/col1/tsm1/backup-pri-dd-3 1274748715008 323228344320 75% 103091 1% /DD/backup-pri-dd-3
ddxxxx-10g:/data/col1/tsm1/backup-pri-dd-4 1274748715008 323228344320 75% 103091 1% /DD/backup-pri-dd-4

Note, this did not require any additional exports from the DataDomain, just creating some directories in it.

This is the devclass for the Datadomain mount

Device Class Name: BACKUP-PRI-DD
Storage Pool Count: 1
Device Type: FILE
Directory: /DD/backup-pri-dd-1,/DD/backup-pri-dd-2,/DD/backup-pri-dd-3,/DD/backup-pri-dd-4

Note - prior to this change the devclass only had Directory: /DD/tsm1/backup-pri-dd





-----Original Message-----
From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Harris, Steven
Sent: Sunday, May 13, 2018 8:39 PM
To: ADSM-L@VM.MARIST.EDU
Subject: [EXTERNAL] Re: ISILON storage/FILE DEVCLASS performance issues

Zoltan

I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual 10Gb links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX only supports NFS3, and as others have pointed out in this forum recently, the stack does not have a good reputation.

I'm finding that the heavy NFS load has other knock on effects, e.g. TSMManager keeps reporting the instance offline when it's very busy as it gets a network error on some of its regular queries, but these work ok when load is light. Also getting a lot of Severed/reconnected sessions. CPU/IO/Paging are not a problem.

Cheers

Steve

Steven Harris
TSM Admin/Consultant
Canberra Australia

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Zoltan Forray
Sent: Saturday, 12 May 2018 1:39 AM
To: ADSM-L@VM.MARIST.EDU
Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues

Folks,

ISP 7.1.7.300 on RHEL 6 10G connectivity

We need some guidance on trying to figure out why ISP/TSM write perform to ISILON storage (via FILE DEVCLASS) is so horrible.

We recently attached 200TB of ISILON storage to this server so we could empty the 36TB of onboard disk drives to move this server to new hardware.

However, per my OS and SAN folks, we are only seeing 1Gbs level of data movement from the ISP server. Doing a regular file copy to this same storage peaks at 10Gbs speeds.

So what, if anything, are we doing wrong when it comes to configuring the storage for ISP to use? Are there some secret controls/settings/options to tell it to use the storage at max-speeds?

We tried changing the Est/Max capacity thinking larger files would reduce the overhead of allocating new pieces constantly. Changed the Mount Limit to a bigger number. Nothing has helped.

Only thing uses the storage right now is migrations from the original disk stgpool.



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/


This message and any attachment is confidential and may be privileged or otherwise protected from disclosure. You should immediately delete the message if you are not the intended recipient. If you have received this email by mistake please delete it from your system; you should not copy the message or disclose its content to anyone.

This electronic communication may contain general financial product advice but should not be relied upon or construed as a recommendation of any financial product. The information has been prepared without taking into account your objectives, financial situation or needs. You should consider the Product Disclosure Statement relating to the financial product and consult your financial adviser before making a decision about whether to acquire, hold or dispose of a financial product.

For further details on the financial product please go to http://www.bt.com.au

Past performance is not a reliable indicator of future performance.
------------------------------------------------------------------------------

The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
This message was imported via the External PhorumMail Module
Zoltan Forray
Re: ISILON storage/FILE DEVCLASS performance issues
May 14, 2018 05:59AM
Very interesting. This supports my idea on how I want to layout the
new/replacement server. The old server is only 16-threads and certainly
could not handle dedup (we can't afford any appliances like DD) since it is
bucking under the current backups traffic. The new server has 72-threads as
well as 100TB internal disk. My idea is to use the fast internal 100TB
disk for inbound traffic and deduping and use the 200TB NFS/ISILON for
nextstoragepool (trying to get completely off 3592-tape storage for onsite
backups). Plus the DB will be on SSD.

Any thoughts on this configuration?

On Sun, May 13, 2018 at 8:38 PM, Harris, Steven <
steven.harris@btfinancialgroup.com> wrote:

> Zoltan
>
> I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual 10Gb
> links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX only
> supports NFS3, and as others have pointed out in this forum recently, the
> stack does not have a good reputation.
>
> I'm finding that the heavy NFS load has other knock on effects, e.g.
> TSMManager keeps reporting the instance offline when it's very busy as it
> gets a network error on some of its regular queries, but these work ok when
> load is light. Also getting a lot of Severed/reconnected sessions.
> CPU/IO/Paging are not a problem.
>
> Cheers
>
> Steve
>
> Steven Harris
> TSM Admin/Consultant
> Canberra Australia
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> Zoltan Forray
> Sent: Saturday, 12 May 2018 1:39 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues
>
> Folks,
>
> ISP 7.1.7.300 on RHEL 6 10G connectivity
>
> We need some guidance on trying to figure out why ISP/TSM write perform to
> ISILON storage (via FILE DEVCLASS) is so horrible.
>
> We recently attached 200TB of ISILON storage to this server so we could
> empty the 36TB of onboard disk drives to move this server to new hardware.
>
> However, per my OS and SAN folks, we are only seeing 1Gbs level of data
> movement from the ISP server. Doing a regular file copy to this same
> storage peaks at 10Gbs speeds.
>
> So what, if anything, are we doing wrong when it comes to configuring the
> storage for ISP to use? Are there some secret controls/settings/options to
> tell it to use the storage at max-speeds?
>
> We tried changing the Est/Max capacity thinking larger files would reduce
> the overhead of allocating new pieces constantly. Changed the Mount Limit
> to a bigger number. Nothing has helped.
>
> Only thing uses the storage right now is migrations from the original disk
> stgpool.
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> Monitor Administrator VMware Administrator Virginia Commonwealth University
> UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu -
> 804-828-4807 Don't be a phishing victim - VCU and other reputable
> organizations will never use email to request that you reply with your
> password, social security number or confidential personal information. For
> more details visit http://phishing.vcu.edu/
>
>
> This message and any attachment is confidential and may be privileged or
> otherwise protected from disclosure. You should immediately delete the
> message if you are not the intended recipient. If you have received this
> email by mistake please delete it from your system; you should not copy the
> message or disclose its content to anyone.
>
> This electronic communication may contain general financial product advice
> but should not be relied upon or construed as a recommendation of any
> financial product. The information has been prepared without taking into
> account your objectives, financial situation or needs. You should consider
> the Product Disclosure Statement relating to the financial product and
> consult your financial adviser before making a decision about whether to
> acquire, hold or dispose of a financial product.
>
> For further details on the financial product please go to
> http://www.bt.com.au
>
> Past performance is not a reliable indicator of future performance.
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: ISILON storage/FILE DEVCLASS performance issues
May 14, 2018 06:59AM
Do you see consistent NFS throughput, or is it bursty? We've never used
Isilon as storage for TSM, but we have had problems generally with too-low
NFS timeouts causing NFS to back off for too long. You can also see this
problem manifest itself with NFS timeout messages in the kernel log.

On Fri, May 11, 2018 at 11:39:08AM -0400, Zoltan Forray wrote:
> Folks,
>
> ISP 7.1.7.300 on RHEL 6 10G connectivity
>
> We need some guidance on trying to figure out why ISP/TSM write perform to
> ISILON storage (via FILE DEVCLASS) is so horrible.
>
> We recently attached 200TB of ISILON storage to this server so we could
> empty the 36TB of onboard disk drives to move this server to new hardware.
>
> However, per my OS and SAN folks, we are only seeing 1Gbs level of data
> movement from the ISP server. Doing a regular file copy to this same
> storage peaks at 10Gbs speeds.
>
> So what, if anything, are we doing wrong when it comes to configuring the
> storage for ISP to use? Are there some secret controls/settings/options to
> tell it to use the storage at max-speeds?
>
> We tried changing the Est/Max capacity thinking larger files would reduce
> the overhead of allocating new pieces constantly. Changed the Mount Limit
> to a bigger number. Nothing has helped.
>
> Only thing uses the storage right now is migrations from the original disk
> stgpool.
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: ISILON storage/FILE DEVCLASS performance issues
May 14, 2018 06:59AM
This sounds pretty good to me. If you can, I would boost your NFS thread
count past the number of CPUs that you have, since a lot of NFS is just
waiting for the disks to respond. You still need a thread for that, but it
won't consume much CPU.

On Mon, May 14, 2018 at 08:27:27AM -0400, Zoltan Forray wrote:
> Very interesting. This supports my idea on how I want to layout the
> new/replacement server. The old server is only 16-threads and certainly
> could not handle dedup (we can't afford any appliances like DD) since it is
> bucking under the current backups traffic. The new server has 72-threads as
> well as 100TB internal disk. My idea is to use the fast internal 100TB
> disk for inbound traffic and deduping and use the 200TB NFS/ISILON for
> nextstoragepool (trying to get completely off 3592-tape storage for onsite
> backups). Plus the DB will be on SSD.
>
> Any thoughts on this configuration?
>
> On Sun, May 13, 2018 at 8:38 PM, Harris, Steven <
> steven.harris@btfinancialgroup.com> wrote:
>
> > Zoltan
> >
> > I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual 10Gb
> > links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX only
> > supports NFS3, and as others have pointed out in this forum recently, the
> > stack does not have a good reputation.
> >
> > I'm finding that the heavy NFS load has other knock on effects, e.g.
> > TSMManager keeps reporting the instance offline when it's very busy as it
> > gets a network error on some of its regular queries, but these work ok when
> > load is light. Also getting a lot of Severed/reconnected sessions.
> > CPU/IO/Paging are not a problem.
> >
> > Cheers
> >
> > Steve
> >
> > Steven Harris
> > TSM Admin/Consultant
> > Canberra Australia
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> > Zoltan Forray
> > Sent: Saturday, 12 May 2018 1:39 AM
> > To: ADSM-L@VM.MARIST.EDU
> > Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues
> >
> > Folks,
> >
> > ISP 7.1.7.300 on RHEL 6 10G connectivity
> >
> > We need some guidance on trying to figure out why ISP/TSM write perform to
> > ISILON storage (via FILE DEVCLASS) is so horrible.
> >
> > We recently attached 200TB of ISILON storage to this server so we could
> > empty the 36TB of onboard disk drives to move this server to new hardware.
> >
> > However, per my OS and SAN folks, we are only seeing 1Gbs level of data
> > movement from the ISP server. Doing a regular file copy to this same
> > storage peaks at 10Gbs speeds.
> >
> > So what, if anything, are we doing wrong when it comes to configuring the
> > storage for ISP to use? Are there some secret controls/settings/options to
> > tell it to use the storage at max-speeds?
> >
> > We tried changing the Est/Max capacity thinking larger files would reduce
> > the overhead of allocating new pieces constantly. Changed the Mount Limit
> > to a bigger number. Nothing has helped.
> >
> > Only thing uses the storage right now is migrations from the original disk
> > stgpool.
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> > Monitor Administrator VMware Administrator Virginia Commonwealth University
> > UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu -
> > 804-828-4807 Don't be a phishing victim - VCU and other reputable
> > organizations will never use email to request that you reply with your
> > password, social security number or confidential personal information. For
> > more details visit http://phishing.vcu.edu/
> >
> >
> > This message and any attachment is confidential and may be privileged or
> > otherwise protected from disclosure. You should immediately delete the
> > message if you are not the intended recipient. If you have received this
> > email by mistake please delete it from your system; you should not copy the
> > message or disclose its content to anyone.
> >
> > This electronic communication may contain general financial product advice
> > but should not be relied upon or construed as a recommendation of any
> > financial product. The information has been prepared without taking into
> > account your objectives, financial situation or needs. You should consider
> > the Product Disclosure Statement relating to the financial product and
> > consult your financial adviser before making a decision about whether to
> > acquire, hold or dispose of a financial product.
> >
> > For further details on the financial product please go to
> > http://www.bt.com.au
> >
> > Past performance is not a reliable indicator of future performance.
> >
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Steven Harris
Re: ISILON storage/FILE DEVCLASS performance issues
May 14, 2018 06:59AM
Thanks for the ideas. Much appreciated. I also looked at some AIX stats
today and they have pointed me at a few hypervisor tuning options.

Cheers

Steve.

On Mon, 14 May 2018, 23:32 Skylar Thompson, <skylar2@uw.edu> wrote:

> This sounds pretty good to me. If you can, I would boost your NFS thread
> count past the number of CPUs that you have, since a lot of NFS is just
> waiting for the disks to respond. You still need a thread for that, but it
> won't consume much CPU.
>
> On Mon, May 14, 2018 at 08:27:27AM -0400, Zoltan Forray wrote:
> > Very interesting. This supports my idea on how I want to layout the
> > new/replacement server. The old server is only 16-threads and certainly
> > could not handle dedup (we can't afford any appliances like DD) since it
> is
> > bucking under the current backups traffic. The new server has 72-threads
> as
> > well as 100TB internal disk. My idea is to use the fast internal 100TB
> > disk for inbound traffic and deduping and use the 200TB NFS/ISILON for
> > nextstoragepool (trying to get completely off 3592-tape storage for
> onsite
> > backups). Plus the DB will be on SSD.
> >
> > Any thoughts on this configuration?
> >
> > On Sun, May 13, 2018 at 8:38 PM, Harris, Steven <
> > steven.harris@btfinancialgroup.com> wrote:
> >
> > > Zoltan
> > >
> > > I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual
> 10Gb
> > > links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX
> only
> > > supports NFS3, and as others have pointed out in this forum recently,
> the
> > > stack does not have a good reputation.
> > >
> > > I'm finding that the heavy NFS load has other knock on effects, e.g.
> > > TSMManager keeps reporting the instance offline when it's very busy as
> it
> > > gets a network error on some of its regular queries, but these work ok
> when
> > > load is light. Also getting a lot of Severed/reconnected sessions.
> > > CPU/IO/Paging are not a problem.
> > >
> > > Cheers
> > >
> > > Steve
> > >
> > > Steven Harris
> > > TSM Admin/Consultant
> > > Canberra Australia
> > >
> > > -----Original Message-----
> > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf
> Of
> > > Zoltan Forray
> > > Sent: Saturday, 12 May 2018 1:39 AM
> > > To: ADSM-L@VM.MARIST.EDU
> > > Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues
> > >
> > > Folks,
> > >
> > > ISP 7.1.7.300 on RHEL 6 10G connectivity
> > >
> > > We need some guidance on trying to figure out why ISP/TSM write
> perform to
> > > ISILON storage (via FILE DEVCLASS) is so horrible.
> > >
> > > We recently attached 200TB of ISILON storage to this server so we could
> > > empty the 36TB of onboard disk drives to move this server to new
> hardware.
> > >
> > > However, per my OS and SAN folks, we are only seeing 1Gbs level of data
> > > movement from the ISP server. Doing a regular file copy to this same
> > > storage peaks at 10Gbs speeds.
> > >
> > > So what, if anything, are we doing wrong when it comes to configuring
> the
> > > storage for ISP to use? Are there some secret
> controls/settings/options to
> > > tell it to use the storage at max-speeds?
> > >
> > > We tried changing the Est/Max capacity thinking larger files would
> reduce
> > > the overhead of allocating new pieces constantly. Changed the Mount
> Limit
> > > to a bigger number. Nothing has helped.
> > >
> > > Only thing uses the storage right now is migrations from the original
> disk
> > > stgpool.
> > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> > > Monitor Administrator VMware Administrator Virginia Commonwealth
> University
> > > UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu -
> > > 804-828-4807 Don't be a phishing victim - VCU and other reputable
> > > organizations will never use email to request that you reply with your
> > > password, social security number or confidential personal information.
> For
> > > more details visit http://phishing.vcu.edu/
> > >
> > >
> > > This message and any attachment is confidential and may be privileged
> or
> > > otherwise protected from disclosure. You should immediately delete the
> > > message if you are not the intended recipient. If you have received
> this
> > > email by mistake please delete it from your system; you should not
> copy the
> > > message or disclose its content to anyone.
> > >
> > > This electronic communication may contain general financial product
> advice
> > > but should not be relied upon or construed as a recommendation of any
> > > financial product. The information has been prepared without taking
> into
> > > account your objectives, financial situation or needs. You should
> consider
> > > the Product Disclosure Statement relating to the financial product and
> > > consult your financial adviser before making a decision about whether
> to
> > > acquire, hold or dispose of a financial product.
> > >
> > > For further details on the financial product please go to
> > > http://www.bt.com.au
> > >
> > > Past performance is not a reliable indicator of future performance.
> > >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray@vcu.edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit http://phishing.vcu.edu/
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>
This message was imported via the External PhorumMail Module
Zoltan Forray
Re: ISILON storage/FILE DEVCLASS performance issues
May 14, 2018 07:59AM
We did some quick research and "NFS thread" controls don't apply in our
situation and can't be set. Or are you referring to the mountlimit value
for the devclass?

On Mon, May 14, 2018 at 9:31 AM, Skylar Thompson <skylar2@uw.edu> wrote:

> This sounds pretty good to me. If you can, I would boost your NFS thread
> count past the number of CPUs that you have, since a lot of NFS is just
> waiting for the disks to respond. You still need a thread for that, but it
> won't consume much CPU.
>
> On Mon, May 14, 2018 at 08:27:27AM -0400, Zoltan Forray wrote:
> > Very interesting. This supports my idea on how I want to layout the
> > new/replacement server. The old server is only 16-threads and certainly
> > could not handle dedup (we can't afford any appliances like DD) since it
> is
> > bucking under the current backups traffic. The new server has 72-threads
> as
> > well as 100TB internal disk. My idea is to use the fast internal 100TB
> > disk for inbound traffic and deduping and use the 200TB NFS/ISILON for
> > nextstoragepool (trying to get completely off 3592-tape storage for
> onsite
> > backups). Plus the DB will be on SSD.
> >
> > Any thoughts on this configuration?
> >
> > On Sun, May 13, 2018 at 8:38 PM, Harris, Steven <
> > steven.harris@btfinancialgroup.com> wrote:
> >
> > > Zoltan
> > >
> > > I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual
> 10Gb
> > > links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX
> only
> > > supports NFS3, and as others have pointed out in this forum recently,
> the
> > > stack does not have a good reputation.
> > >
> > > I'm finding that the heavy NFS load has other knock on effects, e.g.
> > > TSMManager keeps reporting the instance offline when it's very busy as
> it
> > > gets a network error on some of its regular queries, but these work ok
> when
> > > load is light. Also getting a lot of Severed/reconnected sessions.
> > > CPU/IO/Paging are not a problem.
> > >
> > > Cheers
> > >
> > > Steve
> > >
> > > Steven Harris
> > > TSM Admin/Consultant
> > > Canberra Australia
> > >
> > > -----Original Message-----
> > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf
> Of
> > > Zoltan Forray
> > > Sent: Saturday, 12 May 2018 1:39 AM
> > > To: ADSM-L@VM.MARIST.EDU
> > > Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues
> > >
> > > Folks,
> > >
> > > ISP 7.1.7.300 on RHEL 6 10G connectivity
> > >
> > > We need some guidance on trying to figure out why ISP/TSM write
> perform to
> > > ISILON storage (via FILE DEVCLASS) is so horrible.
> > >
> > > We recently attached 200TB of ISILON storage to this server so we could
> > > empty the 36TB of onboard disk drives to move this server to new
> hardware.
> > >
> > > However, per my OS and SAN folks, we are only seeing 1Gbs level of data
> > > movement from the ISP server. Doing a regular file copy to this same
> > > storage peaks at 10Gbs speeds.
> > >
> > > So what, if anything, are we doing wrong when it comes to configuring
> the
> > > storage for ISP to use? Are there some secret
> controls/settings/options to
> > > tell it to use the storage at max-speeds?
> > >
> > > We tried changing the Est/Max capacity thinking larger files would
> reduce
> > > the overhead of allocating new pieces constantly. Changed the Mount
> Limit
> > > to a bigger number. Nothing has helped.
> > >
> > > Only thing uses the storage right now is migrations from the original
> disk
> > > stgpool.
> > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> > > Monitor Administrator VMware Administrator Virginia Commonwealth
> University
> > > UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu -
> > > 804-828-4807 Don't be a phishing victim - VCU and other reputable
> > > organizations will never use email to request that you reply with your
> > > password, social security number or confidential personal information.
> For
> > > more details visit http://phishing.vcu.edu/
> > >
> > >
> > > This message and any attachment is confidential and may be privileged
> or
> > > otherwise protected from disclosure. You should immediately delete the
> > > message if you are not the intended recipient. If you have received
> this
> > > email by mistake please delete it from your system; you should not
> copy the
> > > message or disclose its content to anyone.
> > >
> > > This electronic communication may contain general financial product
> advice
> > > but should not be relied upon or construed as a recommendation of any
> > > financial product. The information has been prepared without taking
> into
> > > account your objectives, financial situation or needs. You should
> consider
> > > the Product Disclosure Statement relating to the financial product and
> > > consult your financial adviser before making a decision about whether
> to
> > > acquire, hold or dispose of a financial product.
> > >
> > > For further details on the financial product please go to
> > > http://www.bt.com.au
> > >
> > > Past performance is not a reliable indicator of future performance.
> > >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray@vcu.edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit http://phishing.vcu.edu/
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: ISILON storage/FILE DEVCLASS performance issues
May 14, 2018 07:59AM
No, this is an NFS server setting, but I'm not sure that it's tunable on
Isilon. On Linux and Solaris, it defaults to some very low value, which is
fine for sequential I/O but really slows down random I/O. On Linux,
RPCNFSDCOUNT can be tuned from the default of 8 to 512, which is fine as
long as the NFS server is running on dedicated hardware.

On Mon, May 14, 2018 at 10:06:31AM -0400, Zoltan Forray wrote:
> We did some quick research and "NFS thread" controls don't apply in our
> situation and can't be set. Or are you referring to the mountlimit value
> for the devclass?
>
> On Mon, May 14, 2018 at 9:31 AM, Skylar Thompson <skylar2@uw.edu> wrote:
>
> > This sounds pretty good to me. If you can, I would boost your NFS thread
> > count past the number of CPUs that you have, since a lot of NFS is just
> > waiting for the disks to respond. You still need a thread for that, but it
> > won't consume much CPU.
> >
> > On Mon, May 14, 2018 at 08:27:27AM -0400, Zoltan Forray wrote:
> > > Very interesting. This supports my idea on how I want to layout the
> > > new/replacement server. The old server is only 16-threads and certainly
> > > could not handle dedup (we can't afford any appliances like DD) since it
> > is
> > > bucking under the current backups traffic. The new server has 72-threads
> > as
> > > well as 100TB internal disk. My idea is to use the fast internal 100TB
> > > disk for inbound traffic and deduping and use the 200TB NFS/ISILON for
> > > nextstoragepool (trying to get completely off 3592-tape storage for
> > onsite
> > > backups). Plus the DB will be on SSD.
> > >
> > > Any thoughts on this configuration?
> > >
> > > On Sun, May 13, 2018 at 8:38 PM, Harris, Steven <
> > > steven.harris@btfinancialgroup.com> wrote:
> > >
> > > > Zoltan
> > > >
> > > > I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual
> > 10Gb
> > > > links, but can only get ~4000 writes/sec and 120MB/sec throughput. AIX
> > only
> > > > supports NFS3, and as others have pointed out in this forum recently,
> > the
> > > > stack does not have a good reputation.
> > > >
> > > > I'm finding that the heavy NFS load has other knock on effects, e.g.
> > > > TSMManager keeps reporting the instance offline when it's very busy as
> > it
> > > > gets a network error on some of its regular queries, but these work ok
> > when
> > > > load is light. Also getting a lot of Severed/reconnected sessions.
> > > > CPU/IO/Paging are not a problem.
> > > >
> > > > Cheers
> > > >
> > > > Steve
> > > >
> > > > Steven Harris
> > > > TSM Admin/Consultant
> > > > Canberra Australia
> > > >
> > > > -----Original Message-----
> > > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf
> > Of
> > > > Zoltan Forray
> > > > Sent: Saturday, 12 May 2018 1:39 AM
> > > > To: ADSM-L@VM.MARIST.EDU
> > > > Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues
> > > >
> > > > Folks,
> > > >
> > > > ISP 7.1.7.300 on RHEL 6 10G connectivity
> > > >
> > > > We need some guidance on trying to figure out why ISP/TSM write
> > perform to
> > > > ISILON storage (via FILE DEVCLASS) is so horrible.
> > > >
> > > > We recently attached 200TB of ISILON storage to this server so we could
> > > > empty the 36TB of onboard disk drives to move this server to new
> > hardware.
> > > >
> > > > However, per my OS and SAN folks, we are only seeing 1Gbs level of data
> > > > movement from the ISP server. Doing a regular file copy to this same
> > > > storage peaks at 10Gbs speeds.
> > > >
> > > > So what, if anything, are we doing wrong when it comes to configuring
> > the
> > > > storage for ISP to use? Are there some secret
> > controls/settings/options to
> > > > tell it to use the storage at max-speeds?
> > > >
> > > > We tried changing the Est/Max capacity thinking larger files would
> > reduce
> > > > the overhead of allocating new pieces constantly. Changed the Mount
> > Limit
> > > > to a bigger number. Nothing has helped.
> > > >
> > > > Only thing uses the storage right now is migrations from the original
> > disk
> > > > stgpool.
> > > >
> > > >
> > > >
> > > > --
> > > > *Zoltan Forray*
> > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> > > > Monitor Administrator VMware Administrator Virginia Commonwealth
> > University
> > > > UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu -
> > > > 804-828-4807 Don't be a phishing victim - VCU and other reputable
> > > > organizations will never use email to request that you reply with your
> > > > password, social security number or confidential personal information.
> > For
> > > > more details visit http://phishing.vcu.edu/
> > > >
> > > >
> > > > This message and any attachment is confidential and may be privileged
> > or
> > > > otherwise protected from disclosure. You should immediately delete the
> > > > message if you are not the intended recipient. If you have received
> > this
> > > > email by mistake please delete it from your system; you should not
> > copy the
> > > > message or disclose its content to anyone.
> > > >
> > > > This electronic communication may contain general financial product
> > advice
> > > > but should not be relied upon or construed as a recommendation of any
> > > > financial product. The information has been prepared without taking
> > into
> > > > account your objectives, financial situation or needs. You should
> > consider
> > > > the Product Disclosure Statement relating to the financial product and
> > > > consult your financial adviser before making a decision about whether
> > to
> > > > acquire, hold or dispose of a financial product.
> > > >
> > > > For further details on the financial product please go to
> > > > http://www.bt.com.au
> > > >
> > > > Past performance is not a reliable indicator of future performance.
> > > >
> > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > Xymon Monitor Administrator
> > > VMware Administrator
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > www.ucc.vcu.edu
> > > zforray@vcu.edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more details
> > > visit http://phishing.vcu.edu/
> >
> > --
> > -- Skylar Thompson (skylar2@u.washington.edu)
> > -- Genome Sciences Department, System Administrator
> > -- Foege Building S046, (206)-685-7354
> > -- University of Washington School of Medicine
> >
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Sergio O. Fuentes
Re: ISILON storage/FILE DEVCLASS performance issues
May 17, 2018 11:59AM
If you're using your Isilon for NFS backend, I would make the following
recommendations:
* Use multiple mountpoints to the Isilon from your TSM server (e.g. Isilon
Client). I would use IP addresses or direct IP connection to the
underlying Isilon nodes to spread the load out against the entire Isilon
cluster (manually).
* Bypass SmartConnect, if that's a licensed thing for you. Don't worry
about failing over your mount points using smartconnect for this. In the
worst case emergencies, you're only impacting backups processing and can
easily reconfigure the mountpoint manually in case of a node outage. If
it's that important, then use a smartconnect policy that makes sense for
TSM deployments (probably round-robin, not one of those fancier
load-balancing options).
* Use many mount points and directories on the TSM server in the devclass.
I have 16 filesystems (aka mounts) per each TSM server devclass.
* Use subdirectories, as suggested in a earlier post, not whole new
exports, although that should work too, to separate the mount points so you
don't run into NFS client locking issues.
* Predefine your file volume sizes. I think directory container pools
might be a better option to use in the future, but that forces you into
dedup, IIRC.

Thanks!
SF





On Mon, May 14, 2018 at 10:25 AM, Skylar Thompson <skylar2@uw.edu> wrote:

> No, this is an NFS server setting, but I'm not sure that it's tunable on
> Isilon. On Linux and Solaris, it defaults to some very low value, which is
> fine for sequential I/O but really slows down random I/O. On Linux,
> RPCNFSDCOUNT can be tuned from the default of 8 to 512, which is fine as
> long as the NFS server is running on dedicated hardware.
>
> On Mon, May 14, 2018 at 10:06:31AM -0400, Zoltan Forray wrote:
> > We did some quick research and "NFS thread" controls don't apply in our
> > situation and can't be set. Or are you referring to the mountlimit value
> > for the devclass?
> >
> > On Mon, May 14, 2018 at 9:31 AM, Skylar Thompson <skylar2@uw.edu> wrote:
> >
> > > This sounds pretty good to me. If you can, I would boost your NFS
> thread
> > > count past the number of CPUs that you have, since a lot of NFS is just
> > > waiting for the disks to respond. You still need a thread for that,
> but it
> > > won't consume much CPU.
> > >
> > > On Mon, May 14, 2018 at 08:27:27AM -0400, Zoltan Forray wrote:
> > > > Very interesting. This supports my idea on how I want to layout the
> > > > new/replacement server. The old server is only 16-threads and
> certainly
> > > > could not handle dedup (we can't afford any appliances like DD)
> since it
> > > is
> > > > bucking under the current backups traffic. The new server has
> 72-threads
> > > as
> > > > well as 100TB internal disk. My idea is to use the fast internal
> 100TB
> > > > disk for inbound traffic and deduping and use the 200TB NFS/ISILON
> for
> > > > nextstoragepool (trying to get completely off 3592-tape storage for
> > > onsite
> > > > backups). Plus the DB will be on SSD.
> > > >
> > > > Any thoughts on this configuration?
> > > >
> > > > On Sun, May 13, 2018 at 8:38 PM, Harris, Steven <
> > > > steven.harris@btfinancialgroup.com> wrote:
> > > >
> > > > > Zoltan
> > > > >
> > > > > I have a similar issue TSM 7.1.1.300 AIX -> Data Domain. Have dual
> > > 10Gb
> > > > > links, but can only get ~4000 writes/sec and 120MB/sec throughput.
> AIX
> > > only
> > > > > supports NFS3, and as others have pointed out in this forum
> recently,
> > > the
> > > > > stack does not have a good reputation.
> > > > >
> > > > > I'm finding that the heavy NFS load has other knock on effects,
> e.g.
> > > > > TSMManager keeps reporting the instance offline when it's very
> busy as
> > > it
> > > > > gets a network error on some of its regular queries, but these
> work ok
> > > when
> > > > > load is light. Also getting a lot of Severed/reconnected sessions.
> > > > > CPU/IO/Paging are not a problem.
> > > > >
> > > > > Cheers
> > > > >
> > > > > Steve
> > > > >
> > > > > Steven Harris
> > > > > TSM Admin/Consultant
> > > > > Canberra Australia
> > > > >
> > > > > -----Original Message-----
> > > > > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On
> Behalf
> > > Of
> > > > > Zoltan Forray
> > > > > Sent: Saturday, 12 May 2018 1:39 AM
> > > > > To: ADSM-L@VM.MARIST.EDU
> > > > > Subject: [ADSM-L] ISILON storage/FILE DEVCLASS performance issues
> > > > >
> > > > > Folks,
> > > > >
> > > > > ISP 7.1.7.300 on RHEL 6 10G connectivity
> > > > >
> > > > > We need some guidance on trying to figure out why ISP/TSM write
> > > perform to
> > > > > ISILON storage (via FILE DEVCLASS) is so horrible.
> > > > >
> > > > > We recently attached 200TB of ISILON storage to this server so we
> could
> > > > > empty the 36TB of onboard disk drives to move this server to new
> > > hardware.
> > > > >
> > > > > However, per my OS and SAN folks, we are only seeing 1Gbs level of
> data
> > > > > movement from the ISP server. Doing a regular file copy to this
> same
> > > > > storage peaks at 10Gbs speeds.
> > > > >
> > > > > So what, if anything, are we doing wrong when it comes to
> configuring
> > > the
> > > > > storage for ISP to use? Are there some secret
> > > controls/settings/options to
> > > > > tell it to use the storage at max-speeds?
> > > > >
> > > > > We tried changing the Est/Max capacity thinking larger files would
> > > reduce
> > > > > the overhead of allocating new pieces constantly. Changed the
> Mount
> > > Limit
> > > > > to a bigger number. Nothing has helped.
> > > > >
> > > > > Only thing uses the storage right now is migrations from the
> original
> > > disk
> > > > > stgpool.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Zoltan Forray*
> > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon
> > > > > Monitor Administrator VMware Administrator Virginia Commonwealth
> > > University
> > > > > UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu
> -
> > > > > 804-828-4807 Don't be a phishing victim - VCU and other reputable
> > > > > organizations will never use email to request that you reply with
> your
> > > > > password, social security number or confidential personal
> information.
> > > For
> > > > > more details visit http://phishing.vcu.edu/
> > > > >
> > > > >
> > > > > This message and any attachment is confidential and may be
> privileged
> > > or
> > > > > otherwise protected from disclosure. You should immediately delete
> the
> > > > > message if you are not the intended recipient. If you have received
> > > this
> > > > > email by mistake please delete it from your system; you should not
> > > copy the
> > > > > message or disclose its content to anyone.
> > > > >
> > > > > This electronic communication may contain general financial product
> > > advice
> > > > > but should not be relied upon or construed as a recommendation of
> any
> > > > > financial product. The information has been prepared without taking
> > > into
> > > > > account your objectives, financial situation or needs. You should
> > > consider
> > > > > the Product Disclosure Statement relating to the financial product
> and
> > > > > consult your financial adviser before making a decision about
> whether
> > > to
> > > > > acquire, hold or dispose of a financial product.
> > > > >
> > > > > For further details on the financial product please go to
> > > > > http://www.bt.com.au
> > > > >
> > > > > Past performance is not a reliable indicator of future performance.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Zoltan Forray*
> > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > Xymon Monitor Administrator
> > > > VMware Administrator
> > > > Virginia Commonwealth University
> > > > UCC/Office of Technology Services
> > > > www.ucc.vcu.edu
> > > > zforray@vcu.edu - 804-828-4807
> > > > Don't be a phishing victim - VCU and other reputable organizations
> will
> > > > never use email to request that you reply with your password, social
> > > > security number or confidential personal information. For more
> details
> > > > visit http://phishing.vcu.edu/
> > >
> > > --
> > > -- Skylar Thompson (skylar2@u.washington.edu)
> > > -- Genome Sciences Department, System Administrator
> > > -- Foege Building S046, (206)-685-7354
> > > -- University of Washington School of Medicine
> > >
> >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray@vcu.edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit http://phishing.vcu.edu/
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login