Welcome! » Log In » Create A New Profile

How to backup ISILON storage

Posted by Zoltan Forray 
Zoltan Forray
How to backup ISILON storage
February 07, 2018 12:59PM
As you recall, we have been trying to figure out an alternative method to
backing up DFS mounted ISILON storage since the current method of 80+
separate nodes accessed via the Web interface of the BA client is going
away. Plus the backups are taking soooooo long, we have to determine a
better way.

So, doing some digging, one solution that seems to be touted is using
NDMP.

We have absolutely zero experience with NDMP and are looking for some
guidance / cookbook / real-world experiences on how we would use NDMP to
backup ISILON storage (>400TB and hundreds of millions of files) and make
it accessible so someone from a help-desk like environment could handle
file-level restores!

Or if NDMP is the wrong direction, please tell us so.
--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: How to backup ISILON storage
February 07, 2018 01:59PM
Content preview: I have stayed away from NDMP because it seems that it locks
you into a particular vendor - if you use Isilon NDMP for backups, then you
have to use Isilon NDMP for the restore. In a major disaster, I would be
worried about the hassle of procuring compatible hardware/software to do the
restore. We instead divide our Isilon storage up into separate NFS mountpoints/TSM
filespaces and then point the client schedules at them with "-domain='/ifs/dir1
/ifs/dir2'". We backup a 2PB OneFS filesystem in this manner, with ~200 million
active files. [...]

Content analysis details: (0.6 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral)
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
X-Barracuda-Start-Time: 1518039059
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://148.100.49.27:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at marist.edu
X-Barracuda-Scan-Msg-Size: 2463
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47680
Rule breakdown below
pts rule name description
---- ---------------------- --------------------------------------------------

I have stayed away from NDMP because it seems that it locks you into a
particular vendor - if you use Isilon NDMP for backups, then you have to
use Isilon NDMP for the restore. In a major disaster, I would be worried
about the hassle of procuring compatible hardware/software to do the
restore. We instead divide our Isilon storage up into separate NFS
mountpoints/TSM filespaces and then point the client schedules at them with
"-domain='/ifs/dir1 /ifs/dir2'". We backup a 2PB OneFS filesystem in this
manner, with ~200 million active files.

We actually are moving away from Isilon for cost reasons though, and moving
towards GPFS. mmbackup removes a lot of the workload division complexity,
though adds other complexity at the same time. That said, it just invokes
dsmc behind the scenes, which means that we can restore our Isilon backups
to GPFS, and vice versa.

On Wed, Feb 07, 2018 at 03:26:02PM -0500, Zoltan Forray wrote:
> As you recall, we have been trying to figure out an alternative method to
> backing up DFS mounted ISILON storage since the current method of 80+
> separate nodes accessed via the Web interface of the BA client is going
> away. Plus the backups are taking soooooo long, we have to determine a
> better way.
>
> So, doing some digging, one solution that seems to be touted is using
> NDMP.
>
> We have absolutely zero experience with NDMP and are looking for some
> guidance / cookbook / real-world experiences on how we would use NDMP to
> backup ISILON storage (>400TB and hundreds of millions of files) and make
> it accessible so someone from a help-desk like environment could handle
> file-level restores!
>
> Or if NDMP is the wrong direction, please tell us so.
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Zoltan Forray
Re: How to backup ISILON storage
February 07, 2018 07:59PM
Interesting. As I said, we have no NDMP experience and wasn't aware of the
vendor specific process.

As for your technique, can you elaborate some more? Where is the ISILON
NFS mounted? To the TSM/ISP server? How do you preserve file rights?
When our SAN guy pursued this (NFS) direction, an EMC forum discussion said
it would not work since "NFS TSM backup would only backup the POSIX
permissions and not the NTFS permissions" and since the ISILON is primarily
accessed as DFS, the file attributes/rights is critical!

On Wed, Feb 7, 2018 at 4:30 PM, Skylar Thompson <skylar2@u.washington.edu>
wrote:

> Content preview: I have stayed away from NDMP because it seems that it
> locks
> you into a particular vendor - if you use Isilon NDMP for backups,
> then you
> have to use Isilon NDMP for the restore. In a major disaster, I would
> be
> worried about the hassle of procuring compatible hardware/software to
> do the
> restore. We instead divide our Isilon storage up into separate NFS
> mountpoints/TSM
> filespaces and then point the client schedules at them with
> "-domain='/ifs/dir1
> /ifs/dir2'". We backup a 2PB OneFS filesystem in this manner, with
> ~200 million
> active files. [...]
>
> Content analysis details: (0.6 points, 5.0 required)
>
> pts rule name description
> ---- ---------------------- ------------------------------
> --------------------
> 0.7 SPF_NEUTRAL SPF: sender does not match SPF record
> (neutral)
> -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
> domain
> X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
> X-Barracuda-Start-Time: 1518039059
> X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
> X-Barracuda-URL: https://148.100.49.27:443/cgi-mod/mark.cgi
> X-Virus-Scanned: by bsmtpd at marist.edu
> X-Barracuda-Scan-Msg-Size: 2463
> X-Barracuda-BRTS-Status: 1
> X-Barracuda-Spam-Score: 0.00
> X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
> TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
> X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47680
> Rule breakdown below
> pts rule name description
> ---- ---------------------- ------------------------------
> --------------------
>
> I have stayed away from NDMP because it seems that it locks you into a
> particular vendor - if you use Isilon NDMP for backups, then you have to
> use Isilon NDMP for the restore. In a major disaster, I would be worried
> about the hassle of procuring compatible hardware/software to do the
> restore. We instead divide our Isilon storage up into separate NFS
> mountpoints/TSM filespaces and then point the client schedules at them with
> "-domain='/ifs/dir1 /ifs/dir2'". We backup a 2PB OneFS filesystem in this
> manner, with ~200 million active files.
>
> We actually are moving away from Isilon for cost reasons though, and moving
> towards GPFS. mmbackup removes a lot of the workload division complexity,
> though adds other complexity at the same time. That said, it just invokes
> dsmc behind the scenes, which means that we can restore our Isilon backups
> to GPFS, and vice versa.
>
> On Wed, Feb 07, 2018 at 03:26:02PM -0500, Zoltan Forray wrote:
> > As you recall, we have been trying to figure out an alternative method to
> > backing up DFS mounted ISILON storage since the current method of 80+
> > separate nodes accessed via the Web interface of the BA client is going
> > away. Plus the backups are taking soooooo long, we have to determine a
> > better way.
> >
> > So, doing some digging, one solution that seems to be touted is using
> > NDMP.
> >
> > We have absolutely zero experience with NDMP and are looking for some
> > guidance / cookbook / real-world experiences on how we would use NDMP to
> > backup ISILON storage (>400TB and hundreds of millions of files) and make
> > it accessible so someone from a help-desk like environment could handle
> > file-level restores!
> >
> > Or if NDMP is the wrong direction, please tell us so.
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator
> > VMware Administrator
> > Virginia Commonwealth University
> > UCC/Office of Technology Services
> > www.ucc.vcu.edu
> > zforray@vcu.edu - 804-828-4807
> > Don't be a phishing victim - VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details
> > visit http://phishing.vcu.edu/
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Remco Post
Re: How to backup ISILON storage
February 08, 2018 01:59AM
> On 7 Feb 2018, at 21:26, Zoltan Forray <zforray@VCU.EDU> wrote:
>
> As you recall, we have been trying to figure out an alternative method to
> backing up DFS mounted ISILON storage since the current method of 80+
> separate nodes accessed via the Web interface of the BA client is going
> away. Plus the backups are taking soooooo long, we have to determine a
> better way.
>
> So, doing some digging, one solution that seems to be touted is using
> NDMP.
>
> We have absolutely zero experience with NDMP and are looking for some
> guidance / cookbook / real-world experiences on how we would use NDMP to
> backup ISILON storage (>400TB and hundreds of millions of files) and make
> it accessible so someone from a help-desk like environment could handle
> file-level restores!

I don’t like TSM NDMP one bit, and I guess it’s no worse than any of the other backup vendors’ implementations, because NDMP is just what it is, and that is not much. I would seriously look into some form of async replication native to ISILON, something like netapp's snapvault, first. Yes that requires a a huge pile of disk just for backup, but it will probably be worth it. Even if the investment is quite high. Don’t forget with TSM terabyte licenses you’ll be paying a lot (a huge lot!) to IBM for your NDMP backups.

You can basically NDMP via LAN and via SAN. The latter has the disadvantage that the TSM server running the backups must be the library manager for those tape drives. I would have loved to see that IBM would make NDMP and Library Managers play nice, but alas… NDMP via LAN allows you to use normal disk and tape based storage pools, via SAN you’ll need to create a separate tape pool in the right format (ndmpdump). Also, you can’t run copy storage pool on those is you use SAN. On 8.1.2 and higher (if you dare go there) you could even use directory containers.

The current customer has NAS systems which share directories (called virtual volumes) rather than separate file systems. To be able to make a more granular backup/restore they use virtualfsmappings in TSM. This works surprisingly well. Now a huge NAS file system becomes (usually) a far more manageable directory. So not 200 TB in one huge lump to backup, but mostly directories of under 1 TB. The backups are slow, but on average manageable. We have a few exceptions that we backup via the share because they are just too big to manage via NDMP. Problem with NDMP is that if (with TSM 8.1) a single transaction spans more than 90% of the active log, the transaction gets killed by TSM. This is on average a good thing, but that makes the combination of a busy TSM server with loads of files and NDMP not a happy one, at least not for those few huge virtual volumes.

So basically:

- look at other solutions (snapvault or whatever it’s called for your NAS)
- then again look at those solutions
- virtualfsmappings might make things more manageable if you decide to go with TSM anyway
- SAN and LAN both have disadvantages, neither one is perfect
- maybe a dedicated TSM instance to avoid issues with long running ndmp dumps

>
> Or if NDMP is the wrong direction, please tell us so.
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--

Met vriendelijke groeten/Kind Regards,

Remco Post
r.post@plcs.nl
+31 6 248 21 622
This message was imported via the External PhorumMail Module
Abbott, Joseph
Re: How to backup ISILON storage
February 08, 2018 03:59AM
I agree with Remco 100%.
If you can stay away from NDMP.
We have a large Isilon environment which we backup with TSM/NDMP. It run very long, is absolutely horrific for restores.
We are making the switch over to Isilon snapshots and replication both native to the Isilon. These solutions outperform TSM/NDMP tenfold.

JoeA

Joseph Abbott, Tivoli Storage Manager Architect CDP
Partners Healthcare AR- 12W60.03
ITS Server & Storage Engineering
Office: 857 -282-3681| Cell:617-633-8471 | Pager: 36364
Need assistance for a non-urgent issue? Open a Service Desk ticket online.

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Remco Post
Sent: Thursday, February 08, 2018 4:17 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] How to backup ISILON storage

> On 7 Feb 2018, at 21:26, Zoltan Forray <zforray@VCU.EDU> wrote:
>
> As you recall, we have been trying to figure out an alternative method
> to backing up DFS mounted ISILON storage since the current method of
> 80+ separate nodes accessed via the Web interface of the BA client is
> going away. Plus the backups are taking soooooo long, we have to
> determine a better way.
>
> So, doing some digging, one solution that seems to be touted is using
> NDMP.
>
> We have absolutely zero experience with NDMP and are looking for some
> guidance / cookbook / real-world experiences on how we would use NDMP
> to backup ISILON storage (>400TB and hundreds of millions of files)
> and make it accessible so someone from a help-desk like environment
> could handle file-level restores!

I don’t like TSM NDMP one bit, and I guess it’s no worse than any of the other backup vendors’ implementations, because NDMP is just what it is, and that is not much. I would seriously look into some form of async replication native to ISILON, something like netapp's snapvault, first. Yes that requires a a huge pile of disk just for backup, but it will probably be worth it. Even if the investment is quite high. Don’t forget with TSM terabyte licenses you’ll be paying a lot (a huge lot!) to IBM for your NDMP backups.

You can basically NDMP via LAN and via SAN. The latter has the disadvantage that the TSM server running the backups must be the library manager for those tape drives. I would have loved to see that IBM would make NDMP and Library Managers play nice, but alas… NDMP via LAN allows you to use normal disk and tape based storage pools, via SAN you’ll need to create a separate tape pool in the right format (ndmpdump). Also, you can’t run copy storage pool on those is you use SAN. On 8.1.2 and higher (if you dare go there) you could even use directory containers.

The current customer has NAS systems which share directories (called virtual volumes) rather than separate file systems. To be able to make a more granular backup/restore they use virtualfsmappings in TSM. This works surprisingly well. Now a huge NAS file system becomes (usually) a far more manageable directory. So not 200 TB in one huge lump to backup, but mostly directories of under 1 TB. The backups are slow, but on average manageable. We have a few exceptions that we backup via the share because they are just too big to manage via NDMP. Problem with NDMP is that if (with TSM 8.1) a single transaction spans more than 90% of the active log, the transaction gets killed by TSM. This is on average a good thing, but that makes the combination of a busy TSM server with loads of files and NDMP not a happy one, at least not for those few huge virtual volumes.

So basically:

- look at other solutions (snapvault or whatever it’s called for your NAS)
- then again look at those solutions
- virtualfsmappings might make things more manageable if you decide to go with TSM anyway
- SAN and LAN both have disadvantages, neither one is perfect
- maybe a dedicated TSM instance to avoid issues with long running ndmp dumps

>
> Or if NDMP is the wrong direction, please tell us so.
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> Monitor Administrator VMware Administrator Virginia Commonwealth
> University UCC/Office of Technology Services www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and
> other reputable organizations will never use email to request that you
> reply with your password, social security number or confidential
> personal information. For more details visit http://phishing.vcu.edu/

--

Met vriendelijke groeten/Kind Regards,

Remco Post
r.post@plcs.nl
+31 6 248 21 622


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.
This message was imported via the External PhorumMail Module
Rhodes, Richard L.
Re: [EXTERNAL] Re: How to backup ISILON storage
February 08, 2018 04:59AM
> I would seriously look into some form of async replication native to ISILON, something like netapp's snapvault, first.

We are just just finishing the first phase of implementing a Isilon NAS, which consolidated dozens of Windows servers. One of the reasons (by no means all of them) of going to the NAS was backups and getting out from under TSM. We setup to use Isilon SyncIQ (replication) and SnapIQ (snapshots) for backups. So far it's worked very well. These features cost $$$ in terms of licensing, disk space and network bandwidth. Besides getting the money, the snapshots require you to guestimate the change rate and retention of your data. Right-click-restore-previous-version is wonderful!

We also have a document management system that has multiple databases with corresponding CIFS shares. Originally the CIFS shares were just Win servers. We used TSM to back it all up. Again, it hit TSM right at its worst - many millions of little files. We were struggling to get 1 good backup of the CIFS shares per day. When the user required 4 backups per day (and other reasons), we converted the whole thing to NetApp NAS/SAN and used snapshots and replication for backups and DR.



Rick






-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Remco Post
Sent: Thursday, February 8, 2018 4:17 AM
To: ADSM-L@VM.MARIST.EDU
Subject: [EXTERNAL] Re: How to backup ISILON storage

> On 7 Feb 2018, at 21:26, Zoltan Forray <zforray@VCU.EDU> wrote:
>
> As you recall, we have been trying to figure out an alternative method to
> backing up DFS mounted ISILON storage since the current method of 80+
> separate nodes accessed via the Web interface of the BA client is going
> away. Plus the backups are taking soooooo long, we have to determine a
> better way.
>
> So, doing some digging, one solution that seems to be touted is using
> NDMP.
>
> We have absolutely zero experience with NDMP and are looking for some
> guidance / cookbook / real-world experiences on how we would use NDMP to
> backup ISILON storage (>400TB and hundreds of millions of files) and make
> it accessible so someone from a help-desk like environment could handle
> file-level restores!

I don’t like TSM NDMP one bit, and I guess it’s no worse than any of the other backup vendors’ implementations, because NDMP is just what it is, and that is not much. I would seriously look into some form of async replication native to ISILON, something like netapp's snapvault, first. Yes that requires a a huge pile of disk just for backup, but it will probably be worth it. Even if the investment is quite high. Don’t forget with TSM terabyte licenses you’ll be paying a lot (a huge lot!) to IBM for your NDMP backups.

You can basically NDMP via LAN and via SAN. The latter has the disadvantage that the TSM server running the backups must be the library manager for those tape drives. I would have loved to see that IBM would make NDMP and Library Managers play nice, but alas… NDMP via LAN allows you to use normal disk and tape based storage pools, via SAN you’ll need to create a separate tape pool in the right format (ndmpdump). Also, you can’t run copy storage pool on those is you use SAN. On 8.1.2 and higher (if you dare go there) you could even use directory containers.

The current customer has NAS systems which share directories (called virtual volumes) rather than separate file systems. To be able to make a more granular backup/restore they use virtualfsmappings in TSM. This works surprisingly well. Now a huge NAS file system becomes (usually) a far more manageable directory. So not 200 TB in one huge lump to backup, but mostly directories of under 1 TB. The backups are slow, but on average manageable. We have a few exceptions that we backup via the share because they are just too big to manage via NDMP. Problem with NDMP is that if (with TSM 8.1) a single transaction spans more than 90% of the active log, the transaction gets killed by TSM. This is on average a good thing, but that makes the combination of a busy TSM server with loads of files and NDMP not a happy one, at least not for those few huge virtual volumes.

So basically:

- look at other solutions (snapvault or whatever it’s called for your NAS)
- then again look at those solutions
- virtualfsmappings might make things more manageable if you decide to go with TSM anyway
- SAN and LAN both have disadvantages, neither one is perfect
- maybe a dedicated TSM instance to avoid issues with long running ndmp dumps

>
> Or if NDMP is the wrong direction, please tell us so.
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--

Met vriendelijke groeten/Kind Regards,

Remco Post
r.post@plcs.nl
+31 6 248 21 622
------------------------------------------------------------------------------

The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: How to backup ISILON storage
February 08, 2018 06:59AM
Content preview: We have a pool of 3 1U servers with 10GbE connectivity that
mount /ifs (and various subdirectories) over NFS. Each node has a set of
schedules that has a subset of the mount points added with -domain statements.
If a node fails, we can move those schedules easily over to another node
while it's being repaired. We actually use this same technique for some legacy
Hitachi/BlueARC storage, and it's worked well for that as well. [...]

Content analysis details: (0.6 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral)
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
0.0 IP_LINK_PLUS URI: Dotted-decimal IP address followed by CGI
0.0 NORMAL_HTTP_TO_IP URI: URI host has a public dotted-decimal IPv4
address
X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
X-Barracuda-Start-Time: 1518100638
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://148.100.49.28:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at marist.edu
X-Barracuda-Scan-Msg-Size: 6650
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.80
X-Barracuda-Spam-Status: No, SCORE=0.80 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=BSF_SC7_SA015c, IP_LINK_PLUS, NORMAL_HTTP_TO_IP
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47712
Rule breakdown below
pts rule name description
---- ---------------------- --------------------------------------------------
0.00 NORMAL_HTTP_TO_IP Uses a dotted-decimal IP address in URL
0.00 IP_LINK_PLUS URI: Dotted-decimal IP address followed by CGI
0.80 BSF_SC7_SA015c Custom Rule SA015c

We have a pool of 3 1U servers with 10GbE connectivity that mount /ifs (and
various subdirectories) over NFS. Each node has a set of schedules that
has a subset of the mount points added with -domain statements. If a node
fails, we can move those schedules easily over to another node while it's
being repaired. We actually use this same technique for some legacy
Hitachi/BlueARC storage, and it's worked well for that as well.

We don't use ACLs or anything beyond POSIX ownership and permissions, so
don't have to worry about that complexity. I think life would be much more
complicated if that were a requirement, though I would still try to find a
way to avoid NDMP.

On Wed, Feb 07, 2018 at 10:07:21PM -0500, Zoltan Forray wrote:
> Interesting. As I said, we have no NDMP experience and wasn't aware of the
> vendor specific process.
>
> As for your technique, can you elaborate some more? Where is the ISILON
> NFS mounted? To the TSM/ISP server? How do you preserve file rights?
> When our SAN guy pursued this (NFS) direction, an EMC forum discussion said
> it would not work since "NFS TSM backup would only backup the POSIX
> permissions and not the NTFS permissions" and since the ISILON is primarily
> accessed as DFS, the file attributes/rights is critical!
>
> On Wed, Feb 7, 2018 at 4:30 PM, Skylar Thompson <skylar2@u.washington.edu>
> wrote:
>
> > Content preview: I have stayed away from NDMP because it seems that it
> > locks
> > you into a particular vendor - if you use Isilon NDMP for backups,
> > then you
> > have to use Isilon NDMP for the restore. In a major disaster, I would
> > be
> > worried about the hassle of procuring compatible hardware/software to
> > do the
> > restore. We instead divide our Isilon storage up into separate NFS
> > mountpoints/TSM
> > filespaces and then point the client schedules at them with
> > "-domain='/ifs/dir1
> > /ifs/dir2'". We backup a 2PB OneFS filesystem in this manner, with
> > ~200 million
> > active files. [...]
> >
> > Content analysis details: (0.6 points, 5.0 required)
> >
> > pts rule name description
> > ---- ---------------------- ------------------------------
> > --------------------
> > 0.7 SPF_NEUTRAL SPF: sender does not match SPF record
> > (neutral)
> > -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
> > domain
> > X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
> > X-Barracuda-Start-Time: 1518039059
> > X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
> > X-Barracuda-URL: https://148.100.49.27:443/cgi-mod/mark.cgi
> > X-Virus-Scanned: by bsmtpd at marist.edu
> > X-Barracuda-Scan-Msg-Size: 2463
> > X-Barracuda-BRTS-Status: 1
> > X-Barracuda-Spam-Score: 0.00
> > X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
> > TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
> > X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47680
> > Rule breakdown below
> > pts rule name description
> > ---- ---------------------- ------------------------------
> > --------------------
> >
> > I have stayed away from NDMP because it seems that it locks you into a
> > particular vendor - if you use Isilon NDMP for backups, then you have to
> > use Isilon NDMP for the restore. In a major disaster, I would be worried
> > about the hassle of procuring compatible hardware/software to do the
> > restore. We instead divide our Isilon storage up into separate NFS
> > mountpoints/TSM filespaces and then point the client schedules at them with
> > "-domain='/ifs/dir1 /ifs/dir2'". We backup a 2PB OneFS filesystem in this
> > manner, with ~200 million active files.
> >
> > We actually are moving away from Isilon for cost reasons though, and moving
> > towards GPFS. mmbackup removes a lot of the workload division complexity,
> > though adds other complexity at the same time. That said, it just invokes
> > dsmc behind the scenes, which means that we can restore our Isilon backups
> > to GPFS, and vice versa.
> >
> > On Wed, Feb 07, 2018 at 03:26:02PM -0500, Zoltan Forray wrote:
> > > As you recall, we have been trying to figure out an alternative method to
> > > backing up DFS mounted ISILON storage since the current method of 80+
> > > separate nodes accessed via the Web interface of the BA client is going
> > > away. Plus the backups are taking soooooo long, we have to determine a
> > > better way.
> > >
> > > So, doing some digging, one solution that seems to be touted is using
> > > NDMP.
> > >
> > > We have absolutely zero experience with NDMP and are looking for some
> > > guidance / cookbook / real-world experiences on how we would use NDMP to
> > > backup ISILON storage (>400TB and hundreds of millions of files) and make
> > > it accessible so someone from a help-desk like environment could handle
> > > file-level restores!
> > >
> > > Or if NDMP is the wrong direction, please tell us so.
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > Xymon Monitor Administrator
> > > VMware Administrator
> > > Virginia Commonwealth University
> > > UCC/Office of Technology Services
> > > www.ucc.vcu.edu
> > > zforray@vcu.edu - 804-828-4807
> > > Don't be a phishing victim - VCU and other reputable organizations will
> > > never use email to request that you reply with your password, social
> > > security number or confidential personal information. For more details
> > > visit http://phishing.vcu.edu/
> >
> > --
> > -- Skylar Thompson (skylar2@u.washington.edu)
> > -- Genome Sciences Department, System Administrator
> > -- Foege Building S046, (206)-685-7354
> > -- University of Washington School of Medicine
> >
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray@vcu.edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://phishing.vcu.edu/

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: How to backup ISILON storage
February 08, 2018 06:59AM
Content preview: We briefly looked into doing replication, but trying to convince
our user base (scientists) that they should get several petabytes of disk
that they couldn't directly use would have been a non-starter. At the time
we also "only" had 10Gbps Internet connection, and sync'ing upwards of 50TB/day
would have consumed a substantial part of that uplink. :) [...]

Content analysis details: (0.6 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral)
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
X-Barracuda-Start-Time: 1518100866
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://148.100.49.28:443/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at marist.edu
X-Barracuda-Scan-Msg-Size: 947
X-Barracuda-BRTS-Status: 1
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47712
Rule breakdown below
pts rule name description
---- ---------------------- --------------------------------------------------

We briefly looked into doing replication, but trying to convince our user
base (scientists) that they should get several petabytes of disk that they
couldn't directly use would have been a non-starter. At the time we also
"only" had 10Gbps Internet connection, and sync'ing upwards of 50TB/day
would have consumed a substantial part of that uplink. :)

On Thu, Feb 08, 2018 at 11:38:30AM +0000, Abbott, Joseph wrote:
> I agree with Remco 100%.
> If you can stay away from NDMP.
> We have a large Isilon environment which we backup with TSM/NDMP. It run very long, is absolutely horrific for restores.
> We are making the switch over to Isilon snapshots and replication both native to the Isilon. These solutions outperform TSM/NDMP tenfold.

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Zoltan Forray
Re: How to backup ISILON storage
February 08, 2018 06:59AM
So you don't have any Windows filesystems on the ISILON? You are a purely
Linux/Unix shop?

On Thu, Feb 8, 2018 at 9:41 AM, Skylar Thompson <skylar2@u.washington.edu>
wrote:

> Content preview: We briefly looked into doing replication, but trying to
> convince
> our user base (scientists) that they should get several petabytes of
> disk
> that they couldn't directly use would have been a non-starter. At the
> time
> we also "only" had 10Gbps Internet connection, and sync'ing upwards of
> 50TB/day
> would have consumed a substantial part of that uplink. :) [...]
>
> Content analysis details: (0.6 points, 5.0 required)
>
> pts rule name description
> ---- ---------------------- ------------------------------
> --------------------
> 0.7 SPF_NEUTRAL SPF: sender does not match SPF record
> (neutral)
> -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
> domain
> X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
> X-Barracuda-Start-Time: 1518100866
> X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
> X-Barracuda-URL: https://148.100.49.28:443/cgi-mod/mark.cgi
> X-Virus-Scanned: by bsmtpd at marist.edu
> X-Barracuda-Scan-Msg-Size: 947
> X-Barracuda-BRTS-Status: 1
> X-Barracuda-Spam-Score: 0.00
> X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
> TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
> X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47712
> Rule breakdown below
> pts rule name description
> ---- ---------------------- ------------------------------
> --------------------
>
> We briefly looked into doing replication, but trying to convince our user
> base (scientists) that they should get several petabytes of disk that they
> couldn't directly use would have been a non-starter. At the time we also
> "only" had 10Gbps Internet connection, and sync'ing upwards of 50TB/day
> would have consumed a substantial part of that uplink. :)
>
> On Thu, Feb 08, 2018 at 11:38:30AM +0000, Abbott, Joseph wrote:
> > I agree with Remco 100%.
> > If you can stay away from NDMP.
> > We have a large Isilon environment which we backup with TSM/NDMP. It run
> very long, is absolutely horrific for restores.
> > We are making the switch over to Isilon snapshots and replication both
> native to the Isilon. These solutions outperform TSM/NDMP tenfold.
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Skylar Thompson
Re: How to backup ISILON storage
February 08, 2018 07:59AM
Content preview: We have a few dozen Windows systems, but nothing complex enough
to require more than simple POSIX permissions. Most of those Windows systems
are instrument systems feeding an analysis pipeline and all connect with
a single user account. The regular user accounts just belong to standard UNIX
groups so don't really require ACLs to manage. [...]

Content analysis details: (0.6 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral)
-0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
domain
X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
X-Barracuda-Start-Time: 1518102266
X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
X-Barracuda-URL: https://148.100.49.28:443/cgi-mod/mark.cgi
X-Barracuda-BRTS-Status: 1
X-Virus-Scanned: by bsmtpd at marist.edu
X-Barracuda-Scan-Msg-Size: 1342
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47714
Rule breakdown below
pts rule name description
---- ---------------------- --------------------------------------------------

We have a few dozen Windows systems, but nothing complex enough to require
more than simple POSIX permissions. Most of those Windows systems are
instrument systems feeding an analysis pipeline and all connect with a
single user account. The regular user accounts just belong to standard UNIX
groups so don't really require ACLs to manage.

Most of the systems using the storage are Linux cluster nodes running the
analysis pipeline over NFS.

On Thu, Feb 08, 2018 at 09:44:37AM -0500, Zoltan Forray wrote:
> So you don't have any Windows filesystems on the ISILON? You are a purely
> Linux/Unix shop?
>
> On Thu, Feb 8, 2018 at 9:41 AM, Skylar Thompson <skylar2@u.washington.edu>
> wrote:
>
> > Content preview: We briefly looked into doing replication, but trying to
> > convince
> > our user base (scientists) that they should get several petabytes of
> > disk
> > that they couldn't directly use would have been a non-starter. At the
> > time
> > we also "only" had 10Gbps Internet connection, and sync'ing upwards of
> > 50TB/day
> > would have consumed a substantial part of that uplink. :) [...]

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Zoltan Forray
Re: How to backup ISILON storage
February 08, 2018 08:59AM
That is pretty much what I figured. That method won't work for us since
most of this ISILON data is from Windows desktops, servers, etc. so ACL
information is crucial since DFS is the key.

Thanks for your suggestions. We do currently use 3-standalone Windows
servers to access the data via DFS mounts but they can't handle the
workload and will probably add even more such boxes but reduce the 85+
nodes into smaller, more manageable groupings and use the GUI (via helpdesk
or the like) for restores since the web client is going away.

On Thu, Feb 8, 2018 at 10:04 AM, Skylar Thompson <skylar2@u.washington.edu>
wrote:

> Content preview: We have a few dozen Windows systems, but nothing
> complex enough
> to require more than simple POSIX permissions. Most of those Windows
> systems
> are instrument systems feeding an analysis pipeline and all connect
> with
> a single user account. The regular user accounts just belong to
> standard UNIX
> groups so don't really require ACLs to manage. [...]
>
> Content analysis details: (0.6 points, 5.0 required)
>
> pts rule name description
> ---- ---------------------- ------------------------------
> --------------------
> 0.7 SPF_NEUTRAL SPF: sender does not match SPF record
> (neutral)
> -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay
> domain
> X-Barracuda-Connect: mx.gs.washington.edu[128.208.8.134]
> X-Barracuda-Start-Time: 1518102266
> X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384
> X-Barracuda-URL: https://148.100.49.28:443/cgi-mod/mark.cgi
> X-Barracuda-BRTS-Status: 1
> X-Virus-Scanned: by bsmtpd at marist.edu
> X-Barracuda-Scan-Msg-Size: 1342
> X-Barracuda-Spam-Score: 0.00
> X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of
> TAG_LEVEL=3.5 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=5.5 tests=
> X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.47714
> Rule breakdown below
> pts rule name description
> ---- ---------------------- ------------------------------
> --------------------
>
> We have a few dozen Windows systems, but nothing complex enough to require
> more than simple POSIX permissions. Most of those Windows systems are
> instrument systems feeding an analysis pipeline and all connect with a
> single user account. The regular user accounts just belong to standard UNIX
> groups so don't really require ACLs to manage.
>
> Most of the systems using the storage are Linux cluster nodes running the
> analysis pipeline over NFS.
>
> On Thu, Feb 08, 2018 at 09:44:37AM -0500, Zoltan Forray wrote:
> > So you don't have any Windows filesystems on the ISILON? You are a purely
> > Linux/Unix shop?
> >
> > On Thu, Feb 8, 2018 at 9:41 AM, Skylar Thompson <
> skylar2@u.washington.edu>
> > wrote:
> >
> > > Content preview: We briefly looked into doing replication, but
> trying to
> > > convince
> > > our user base (scientists) that they should get several petabytes
> of
> > > disk
> > > that they couldn't directly use would have been a non-starter. At
> the
> > > time
> > > we also "only" had 10Gbps Internet connection, and sync'ing
> upwards of
> > > 50TB/day
> > > would have consumed a substantial part of that uplink. :) [...]
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>



--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login