Welcome! » Log In » Create A New Profile

Looking for suggestions to deal with large backups not completing in 24-hours: the GWDG solution briefly explained

Posted by Nachtwey, Bjoern 
Hi Zoltan,

i will come back to the approach Jonas mentioned (as I'm the author of that text: thanks to Jonas for doing this ;-) )

the text is in german of course, but the script has some comments in English and will be understandable -- I hope so :-)

the text describes first the problem everybody on this list will know: the treewalk takes more times than we have.
TSM/ISP has some opportunities to speed up, such as "-incrbydate", but they do not work properly.

So for me the only solution is to parallelize the tree walk and do partial incremental backups.
First tried to write it with BASH commands, but multithreading was not easy to implement and second it won't run on windows -- but our largest filers ( 500 TB - 1.2 PB) need to be accessed via CIFS to store the ACL information.
My first steps with PowerShell for the Windows cost lots of time and were disappointing.
Using PERL made everything really easy as it runs on windows with the strawberry perl software and within the script there are only a few if-conditions needed to determine between Linux and Windows.

I did some tests according to the depth or the level of the filetree to dive in:
As the subfolders are of unequal size, diving just below the mount point and parallelize on the folders of this "first level" mostly does not work well, there's (nearly) always one folder taking all the time.
On the other hand diving into all levels will take a certain amount of additional time.

The best performance I do see using 3 to 4 levels and 4 to 6 parallel threads for each node. Due to separating users and for accounting I have several nodes on such large file systems. So in total there are about 20 to 40 streams in parallel.

Rudi Wüst mentioned in my text figured out a p520 server running AIX6 will support up to 2,000 parallel streams, but as mentioned by Grant using an isilon system the filer will be the bottle neck.

As mentioned by Del, you may also test a commercial software "MAGS" by general storage, it can addresses multiple isilon nodes in parallel

If there're any questions -- just ask or have a look on the script:
https://gitlab.gwdg.de/bnachtw/dsmci

// even if the last submit is about 4 month old, the project is still in development ;-)



==> maybe I should update and translate the text from the "GWDG news" to English? Any interest?


Best
Bjørn


p.s.
A Result from the wild (weekly backup of a node from a 343 TB Quantum StorNext File System) :
>>
Process ID : 12988
Path processed : <removed>
-------------------------------------------------
Start time : 2018-07-14 12:00
End time : 2018-07-15 06:07
total processing time : 3d 15h 59m 23s
total wallclock time : 18h 7m 30s
effective speedup : 4.855 using 6 parallel threads
datatransfertime ratio: 3.575 %
-------------------------------------------------
Objects inspected : 92061596
Objects backed up : 9774876
Objects updated : 0
Objects deleted : 0
Objects expired : 7696
Objects failed : 0
Bytes inspected : 52818.242 (GB)
Bytes transferred : 5063.620 (GB)
-------------------------------------------------
Number of Errors : 0
Number of Warnings : 43
# of severe Errors : 0
# Out-of-Space Errors : 0
<<

--------------------------------------------------------------------------------------------------
Bjørn Nachtwey

Arbeitsgruppe "IT-Infrastruktur“
Tel.: +49 551 201-2181, E-Mail: bjoern.nachtwey@gwdg.de
--------------------------------------------------------------------------------------------------
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de
Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: gwdg@gwdg.de
Service-Hotline: Tel.: +49 551 201-1523, E-Mail: support@gwdg.de
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lossau
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen, Handelsregister-Nr. B 598
--------------------------------------------------------------------------------------------------
Zertifiziert nach ISO 9001
--------------------------------------------------------------------------------------------------

-----Ursprüngliche Nachricht-----
Von: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> Im Auftrag von Zoltan Forray
Gesendet: Mittwoch, 11. Juli 2018 13:50
An: ADSM-L@VM.MARIST.EDU
Betreff: Re: [ADSM-L] Looking for suggestions to deal with large backups not completing in 24-hours

I will need to translate to English but I gather it is talking about the RESOURCEUTILZATION / MAXNUMMP values. While we have increased MAXNUMMP to
5 on the server (will try going higher), not sure how much good it would do since the backup schedule uses OBJECTS to point to a specific/single mountpoint/filesystem (see below) but is worth trying to bump the RESOURCEUTILIZATION value on the client even higher...

We have checked the dsminstr.log file and it is spending 92% of the time in PROCESS DIRS (no surprise)

7:46:25 AM SUN : q schedule * ISILON-SOM-SOMADFS1 f=d
Policy Domain Name: DFS
Schedule Name: ISILON-SOM-SOMADFS1
Description: ISILON-SOM-SOMADFS1
Action: Incremental
Subaction:
Options: -subdir=yes
Objects: \\rams.adp.vcu.edu\SOM\TSM\SOMADFS1\*
Priority: 5
Start Date/Time: 12/05/2017 08:30:00
Duration: 1 Hour(s)
Maximum Run Time (Minutes): 0
Schedule Style: Enhanced
Period:
Day of Week: Any
Month: Any
Day of Month: Any
Week of Month: Any
Expiration:
Last Update by (administrator): ZFORRAY
Last Update Date/Time: 01/12/2018 10:30:48
Managing profile:


On Tue, Jul 10, 2018 at 4:06 AM Jansen, Jonas <jansen@itc.rwth-aachen.de>
wrote:

> It is possible to da a parallel backup of file system parts.
> https://www.gwdg.de/documents/20182/27257/GN_11-2016_www.pdf (german)
> have a look on page 10.
>
> ---
> Jonas Jansen
>
> IT Center
> Gruppe: Server & Storage
> Abteilung: Systeme & Betrieb
> RWTH Aachen University
> Seffenter Weg 23
> 52074 Aachen
> Tel: +49 241 80-28784
> Fax: +49 241 80-22134
> jansen@itc.rwth-aachen.de
> www.itc.rwth-aachen.de
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Del
> Hoobler
> Sent: Monday, July 9, 2018 3:29 PM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] Looking for suggestions to deal with large
> backups not completing in 24-hours
>
> They are a 3rd-party partner that offers an integrated Spectrum
> Protect solution for large filer backups.
>
>
> Del
>
> ----------------------------------------------------
>
> "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 07/09/2018
> 09:17:06 AM:
>
> > From: Zoltan Forray <zforray@VCU.EDU>
> > To: ADSM-L@VM.MARIST.EDU
> > Date: 07/09/2018 09:17 AM
> > Subject: Re: Looking for suggestions to deal with large backups not
> > completing in 24-hours Sent by: "ADSM: Dist Stor Manager"
> > <ADSM-L@VM.MARIST.EDU>
> >
> > Thanks Del. Very interesting. Are they a VAR for IBM?
> >
> > Not sure if it would work in the current configuration we are using
> > to
> back
> > up ISILON. I have passed the info on.
> >
> > BTW, FWIW, when I copied/pasted the info, Chrome spell-checker
> red-flagged
> > on "The easy way to incrementally backup billons of objects" (billions).
> > So if you know anybody at the company, please pass it on to them.
> >
> > On Mon, Jul 9, 2018 at 6:51 AM Del Hoobler <hoobler@us.ibm.com> wrote:
> >
> > > Another possible idea is to look at General Storage dsmISI MAGS:
> > >
> > > INVALID URI REMOVED
> >
>
> u=http-3A__www.general-2Dstorage.com_PRODUCTS_products.html&d=DwIBaQ&c
> =jf_ia
> SHvJObTbx-
> >
>
> siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=ofZM7gZ7p5GL1H
> FyHU75
> lwUZLmc_kYAQxroVCZQUCSs&s=25_psxEcE0fvxruxybvMJZzSZv-
> > ach7r-VHXaLNVD_E&e=
> > >
> > >
> > > Del
> > >
> > >
> > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on
> > > 07/05/2018
> > > 02:52:27 PM:
> > >
> > > > From: Zoltan Forray <zforray@VCU.EDU>
> > > > To: ADSM-L@VM.MARIST.EDU
> > > > Date: 07/05/2018 02:53 PM
> > > > Subject: Looking for suggestions to deal with large backups not
> > > > completing in 24-hours Sent by: "ADSM: Dist Stor Manager"
> > > > <ADSM-L@VM.MARIST.EDU>
> > > >
> > > > As I have mentioned in the past, we have gone through large
> migrations
> > > to
> > > > DFS based storage on EMC ISILON hardware. As you may recall, we
> backup
> > > > these DFS mounts (about 90 at last count) using multiple Windows
> servers
> > > > that run multiple ISP nodes (about 30-each) and they access each
> > > > DFS mount/filesystem via -object=\\rams.adp.vcu.edu\departmentname.
> > > >
> > > > This has lead to lots of performance issue with backups and some
> > > > departments are now complain that their backups are running into
> > > > multiple-days in some cases.
> > > >
> > > > One such case in a department with 2-nodes with over 30-million
> objects
> > > for
> > > > each node. In the past, their backups were able to finish
> > > > quicker
> since
> > > > they were accessed via dedicated servers and were able to use
> Journaling
> > > to
> > > > reduce the scan times. Unless things have changed, I believe
> Journling
> > > is
> > > > not an option due to how the files are accessed.
> > > >
> > > > FWIW, average backups are usually <50k files and <200GB once it
> finished
> > > > scanning.....
> > > >
> > > > Also, the idea of HSM/SPACEMANAGEMENT has reared its ugly head
> > > > since
> > > many
> > > > of these objects haven't been accessed in many years old. But as
> > > > I understand it, that won't work either given our current
> configuration.
> > > >
> > > > Given the current DFS configuration (previously CIFS), what can
> > > > we
> do to
> > > > improve backup performance?
> > > >
> > > > So, any-and-all ideas are up for discussion. There is even
> discussion
> > > on
> > > > replacing ISP/TSM due to these issues/limitations.
> > > >
> > > > --
> > > > *Zoltan Forray*
> > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > Xymon Monitor Administrator VMware Administrator Virginia
> > > > Commonwealth University UCC/Office of Technology Services
> > > > www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a
> > > > phishing victim - VCU and other reputable organizations
> will
> > > > never use email to request that you reply with your password,
> > > > social security number or confidential personal information. For
> > > > more
> details
> > > > visit INVALID URI REMOVED
> > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > > siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=5bz_TktY
> > > > 3-
> > > > a432oKYronO-w1z-
> > > > ax8md3tzFqX9nGxoU&s=EudIhVvfUVx4-5UmfJHaRUzHCd7Agwk3Pog8wmEEpdA&
> > > > e=
> > > >
> > >
> >
> >
> > --
> > *Zoltan Forray*
> > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > Xymon Monitor Administrator VMware Administrator Virginia
> > Commonwealth University UCC/Office of Technology Services
> > www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a phishing
> > victim - VCU and other reputable organizations will never use email
> > to request that you reply with your password, social security number
> > or confidential personal information. For more details visit INVALID
> > URI REMOVED
> > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> >
>
> siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=ofZM7gZ7p5GL1H
> FyHU75
> lwUZLmc_kYAQxroVCZQUCSs&s=umTd28h-
> > GlxqSvNShsNIqm8D1PcanVk0HPcP5KTurKw&e=
> >
>


--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Bjørn,

Thank you for the details. As the common consensus is, we need to break-up
the number of directories/files each node processes/scans. Also seem to
need the use of the PROXY NODE process to consolidate access into one
node/client since 5+ nodes will be required to process what is now being
attempted through 1-node.

On Tue, Jul 17, 2018 at 8:05 AM Nachtwey, Bjoern <bjoern.nachtwey@gwdg.de>
wrote:

> Hi Zoltan,
>
> i will come back to the approach Jonas mentioned (as I'm the author of
> that text: thanks to Jonas for doing this ;-) )
>
> the text is in german of course, but the script has some comments in
> English and will be understandable -- I hope so :-)
>
> the text describes first the problem everybody on this list will know: the
> treewalk takes more times than we have.
> TSM/ISP has some opportunities to speed up, such as "-incrbydate", but
> they do not work properly.
>
> So for me the only solution is to parallelize the tree walk and do partial
> incremental backups.
> First tried to write it with BASH commands, but multithreading was not
> easy to implement and second it won't run on windows -- but our largest
> filers ( 500 TB - 1.2 PB) need to be accessed via CIFS to store the ACL
> information.
> My first steps with PowerShell for the Windows cost lots of time and were
> disappointing.
> Using PERL made everything really easy as it runs on windows with the
> strawberry perl software and within the script there are only a few
> if-conditions needed to determine between Linux and Windows.
>
> I did some tests according to the depth or the level of the filetree to
> dive in:
> As the subfolders are of unequal size, diving just below the mount point
> and parallelize on the folders of this "first level" mostly does not work
> well, there's (nearly) always one folder taking all the time.
> On the other hand diving into all levels will take a certain amount of
> additional time.
>
> The best performance I do see using 3 to 4 levels and 4 to 6 parallel
> threads for each node. Due to separating users and for accounting I have
> several nodes on such large file systems. So in total there are about 20 to
> 40 streams in parallel.
>
> Rudi Wüst mentioned in my text figured out a p520 server running AIX6 will
> support up to 2,000 parallel streams, but as mentioned by Grant using an
> isilon system the filer will be the bottle neck.
>
> As mentioned by Del, you may also test a commercial software "MAGS" by
> general storage, it can addresses multiple isilon nodes in parallel
>
> If there're any questions -- just ask or have a look on the script:
> https://gitlab.gwdg.de/bnachtw/dsmci
>
> // even if the last submit is about 4 month old, the project is still in
> development ;-)
>
>
>
> ==> maybe I should update and translate the text from the "GWDG news" to
> English? Any interest?
>
>
> Best
> Bjørn
>
>
> p.s.
> A Result from the wild (weekly backup of a node from a 343 TB Quantum
> StorNext File System) :
> >>
> Process ID : 12988
> Path processed : <removed>
> -------------------------------------------------
> Start time : 2018-07-14 12:00
> End time : 2018-07-15 06:07
> total processing time : 3d 15h 59m 23s
> total wallclock time : 18h 7m 30s
> effective speedup : 4.855 using 6 parallel threads
> datatransfertime ratio: 3.575 %
> -------------------------------------------------
> Objects inspected : 92061596
> Objects backed up : 9774876
> Objects updated : 0
> Objects deleted : 0
> Objects expired : 7696
> Objects failed : 0
> Bytes inspected : 52818.242 (GB)
> Bytes transferred : 5063.620 (GB)
> -------------------------------------------------
> Number of Errors : 0
> Number of Warnings : 43
> # of severe Errors : 0
> # Out-of-Space Errors : 0
> <<
>
> --------------------------------------------------------------------------------------------------
>
> Bjørn Nachtwey
>
> Arbeitsgruppe "IT-Infrastruktur“
> Tel.: +49 551 201-2181, E-Mail: bjoern.nachtwey@gwdg.de
> --------------------------------------------------------------------------------------------------
>
> Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
> Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de
> Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: gwdg@gwdg.de
> Service-Hotline: Tel.: +49 551 201-1523, E-Mail: support@gwdg.de
> Geschäftsführer: Prof. Dr. Ramin Yahyapour
> Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lossau
> Sitz der Gesellschaft: Göttingen
> Registergericht: Göttingen, Handelsregister-Nr. B 598
> --------------------------------------------------------------------------------------------------
>
> Zertifiziert nach ISO 9001
>
> --------------------------------------------------------------------------------------------------
>
> -----Ursprüngliche Nachricht-----
> Von: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> Im Auftrag von Zoltan
> Forray
> Gesendet: Mittwoch, 11. Juli 2018 13:50
> An: ADSM-L@VM.MARIST.EDU
> Betreff: Re: [ADSM-L] Looking for suggestions to deal with large backups
> not completing in 24-hours
>
> I will need to translate to English but I gather it is talking about the
> RESOURCEUTILZATION / MAXNUMMP values. While we have increased MAXNUMMP to
> 5 on the server (will try going higher), not sure how much good it would
> do since the backup schedule uses OBJECTS to point to a specific/single
> mountpoint/filesystem (see below) but is worth trying to bump the
> RESOURCEUTILIZATION value on the client even higher...
>
> We have checked the dsminstr.log file and it is spending 92% of the time
> in PROCESS DIRS (no surprise)
>
> 7:46:25 AM SUN : q schedule * ISILON-SOM-SOMADFS1 f=d
> Policy Domain Name: DFS
> Schedule Name: ISILON-SOM-SOMADFS1
> Description: ISILON-SOM-SOMADFS1
> Action: Incremental
> Subaction:
> Options: -subdir=yes
> Objects: \\rams.adp.vcu.edu\SOM\TSM\SOMADFS1\*
> Priority: 5
> Start Date/Time: 12/05/2017 08:30:00
> Duration: 1 Hour(s)
> Maximum Run Time (Minutes): 0
> Schedule Style: Enhanced
> Period:
> Day of Week: Any
> Month: Any
> Day of Month: Any
> Week of Month: Any
> Expiration:
> Last Update by (administrator): ZFORRAY
> Last Update Date/Time: 01/12/2018 10:30:48
> Managing profile:
>
>
> On Tue, Jul 10, 2018 at 4:06 AM Jansen, Jonas <jansen@itc.rwth-aachen.de>
> wrote:
>
> > It is possible to da a parallel backup of file system parts.
> > https://www.gwdg.de/documents/20182/27257/GN_11-2016_www.pdf (german)
> > have a look on page 10.
> >
> > ---
> > Jonas Jansen
> >
> > IT Center
> > Gruppe: Server & Storage
> > Abteilung: Systeme & Betrieb
> > RWTH Aachen University
> > Seffenter Weg 23
> > 52074 Aachen
> > Tel: +49 241 80-28784
> > Fax: +49 241 80-22134
> > jansen@itc.rwth-aachen.de
> > www.itc.rwth-aachen.de
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Del
> > Hoobler
> > Sent: Monday, July 9, 2018 3:29 PM
> > To: ADSM-L@VM.MARIST.EDU
> > Subject: Re: [ADSM-L] Looking for suggestions to deal with large
> > backups not completing in 24-hours
> >
> > They are a 3rd-party partner that offers an integrated Spectrum
> > Protect solution for large filer backups.
> >
> >
> > Del
> >
> > ----------------------------------------------------
> >
> > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on 07/09/2018
> > 09:17:06 AM:
> >
> > > From: Zoltan Forray <zforray@VCU.EDU>
> > > To: ADSM-L@VM.MARIST.EDU
> > > Date: 07/09/2018 09:17 AM
> > > Subject: Re: Looking for suggestions to deal with large backups not
> > > completing in 24-hours Sent by: "ADSM: Dist Stor Manager"
> > > <ADSM-L@VM.MARIST.EDU>
> > >
> > > Thanks Del. Very interesting. Are they a VAR for IBM?
> > >
> > > Not sure if it would work in the current configuration we are using
> > > to
> > back
> > > up ISILON. I have passed the info on.
> > >
> > > BTW, FWIW, when I copied/pasted the info, Chrome spell-checker
> > red-flagged
> > > on "The easy way to incrementally backup billons of objects"
> (billions).
> > > So if you know anybody at the company, please pass it on to them.
> > >
> > > On Mon, Jul 9, 2018 at 6:51 AM Del Hoobler <hoobler@us.ibm.com> wrote:
> > >
> > > > Another possible idea is to look at General Storage dsmISI MAGS:
> > > >
> > > > INVALID URI REMOVED
> > >
> >
> > u=http-3A__www.general-2Dstorage.com_PRODUCTS_products.html&d=DwIBaQ&c
> > =jf_ia
> > SHvJObTbx-
> > >
> >
> > siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=ofZM7gZ7p5GL1H
> > FyHU75
> > lwUZLmc_kYAQxroVCZQUCSs&s=25_psxEcE0fvxruxybvMJZzSZv-
> > > ach7r-VHXaLNVD_E&e=
> > > >
> > > >
> > > > Del
> > > >
> > > >
> > > > "ADSM: Dist Stor Manager" <ADSM-L@VM.MARIST.EDU> wrote on
> > > > 07/05/2018
> > > > 02:52:27 PM:
> > > >
> > > > > From: Zoltan Forray <zforray@VCU.EDU>
> > > > > To: ADSM-L@VM.MARIST.EDU
> > > > > Date: 07/05/2018 02:53 PM
> > > > > Subject: Looking for suggestions to deal with large backups not
> > > > > completing in 24-hours Sent by: "ADSM: Dist Stor Manager"
> > > > > <ADSM-L@VM.MARIST.EDU>
> > > > >
> > > > > As I have mentioned in the past, we have gone through large
> > migrations
> > > > to
> > > > > DFS based storage on EMC ISILON hardware. As you may recall, we
> > backup
> > > > > these DFS mounts (about 90 at last count) using multiple Windows
> > servers
> > > > > that run multiple ISP nodes (about 30-each) and they access each
> > > > > DFS mount/filesystem via -object=\\rams.adp.vcu.edu
> \departmentname.
> > > > >
> > > > > This has lead to lots of performance issue with backups and some
> > > > > departments are now complain that their backups are running into
> > > > > multiple-days in some cases.
> > > > >
> > > > > One such case in a department with 2-nodes with over 30-million
> > objects
> > > > for
> > > > > each node. In the past, their backups were able to finish
> > > > > quicker
> > since
> > > > > they were accessed via dedicated servers and were able to use
> > Journaling
> > > > to
> > > > > reduce the scan times. Unless things have changed, I believe
> > Journling
> > > > is
> > > > > not an option due to how the files are accessed.
> > > > >
> > > > > FWIW, average backups are usually <50k files and <200GB once it
> > finished
> > > > > scanning.....
> > > > >
> > > > > Also, the idea of HSM/SPACEMANAGEMENT has reared its ugly head
> > > > > since
> > > > many
> > > > > of these objects haven't been accessed in many years old. But as
> > > > > I understand it, that won't work either given our current
> > configuration.
> > > > >
> > > > > Given the current DFS configuration (previously CIFS), what can
> > > > > we
> > do to
> > > > > improve backup performance?
> > > > >
> > > > > So, any-and-all ideas are up for discussion. There is even
> > discussion
> > > > on
> > > > > replacing ISP/TSM due to these issues/limitations.
> > > > >
> > > > > --
> > > > > *Zoltan Forray*
> > > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > > > Xymon Monitor Administrator VMware Administrator Virginia
> > > > > Commonwealth University UCC/Office of Technology Services
> > > > > www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a
> > > > > phishing victim - VCU and other reputable organizations
> > will
> > > > > never use email to request that you reply with your password,
> > > > > social security number or confidential personal information. For
> > > > > more
> > details
> > > > > visit INVALID URI REMOVED
> > > > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > > > > siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=5bz_TktY
> > > > > 3-
> > > > > a432oKYronO-w1z-
> > > > > ax8md3tzFqX9nGxoU&s=EudIhVvfUVx4-5UmfJHaRUzHCd7Agwk3Pog8wmEEpdA&
> > > > > e=
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Zoltan Forray*
> > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > > Xymon Monitor Administrator VMware Administrator Virginia
> > > Commonwealth University UCC/Office of Technology Services
> > > www.ucc.vcu.edu zforray@vcu.edu - 804-828-4807 Don't be a phishing
> > > victim - VCU and other reputable organizations will never use email
> > > to request that you reply with your password, social security number
> > > or confidential personal information. For more details visit INVALID
> > > URI REMOVED
> > > u=http-3A__phishing.vcu.edu_&d=DwIBaQ&c=jf_iaSHvJObTbx-
> > >
> >
> > siA1ZOg&r=0hq2JX5c3TEZNriHEs7Zf7HrkY2fNtONOrEOM8Txvk8&m=ofZM7gZ7p5GL1H
> > FyHU75
> > lwUZLmc_kYAQxroVCZQUCSs&s=umTd28h-
> > > GlxqSvNShsNIqm8D1PcanVk0HPcP5KTurKw&e=
> > >
> >
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon
> Monitor Administrator VMware Administrator Virginia Commonwealth University
> UCC/Office of Technology Services www.ucc.vcu.edu zforray@vcu.edu -
> 804-828-4807 Don't be a phishing victim - VCU and other reputable
> organizations will never use email to request that you reply with your
> password, social security number or confidential personal information. For
> more details visit http://phishing.vcu.edu/
>


--
*Zoltan Forray*
Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
Xymon Monitor Administrator
VMware Administrator
Virginia Commonwealth University
UCC/Office of Technology Services
www.ucc.vcu.edu
zforray@vcu.edu - 804-828-4807
Don't be a phishing victim - VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information. For more details
visit http://phishing.vcu.edu/
This message was imported via the External PhorumMail Module
Hi Zoltan,

OK, i will translate my text as there are some more approaches discussed :-)

breaking up the filesystems in several nodes will work as long as the
nodes are of suffiecient size.

I'm not sure if a PROXY node will solve the problem, because each
"member node" will backup the whole mountpoint. You will need to do
partial incremental backups. I expect you will do this based on folders,
do you?
So, some questions:
1) how will you distribute the folders to the nodes?
2) how will you ensure new folders are processed by one of your "member
nodes"? On our filers many folders are created and deleted, sometimes a
whole bunch every day. So for me, it was no option to maintain the
option file manually. The approach from my script / "MAGS" does this
somehow "automatically".
3) what happens if the folders grew not evenly and all the big ones are
backed up by one of your nodes? (OK you can change the distribution or
even add another node)
4) Are you going to map each backupnode to different nodes of the isilon
cluster to distribute the traffic / workload for the isilon nodes?

best
Bjørn
This message was imported via the External PhorumMail Module
One thing to be aware of with partial incremental backups is the danger of
backing up data multiple times if the mount points are nested. For
instance,

/mnt/backup/some-dir
/mnt/backup/some-dir/another-dir

Under normal operation, a node with DOMAIN set to "/mnt/backup/some-dir
/mnt/backup/some-dir/another-dir" will backup the contents of /mnt/backup/some-dir/another-dir
as a separate filespace, *and also* will backup another-dir as a
subdirectory of the /mnt/backup/some-dir filespace. We reported this as a
bug, and IBM pointed us at this flag that can be passed as a scheduler
option to prevent this:

-TESTFLAG=VMPUNDERNFSENABLED

On Tue, Jul 17, 2018 at 04:12:17PM +0200, Bjrn Nachtwey wrote:
> Hi Zoltan,
>
> OK, i will translate my text as there are some more approaches discussed :-)
>
> breaking up the filesystems in several nodes will work as long as the nodes
> are of suffiecient size.
>
> I'm not sure if a PROXY node will solve the problem, because each "member
> node" will backup the whole mountpoint. You will need to do partial
> incremental backups. I expect you will do this based on folders, do you?
> So, some questions:
> 1) how will you distribute the folders to the nodes?
> 2) how will you ensure new folders are processed by one of your "member
> nodes"? On our filers many folders are created and deleted, sometimes a
> whole bunch every day. So for me, it was no option to maintain the option
> file manually. The approach from my script / "MAGS" does this somehow
> "automatically".
> 3) what happens if the folders grew not evenly and all the big ones are
> backed up by one of your nodes? (OK you can change the distribution or even
> add another node)
> 4) Are you going to map each backupnode to different nodes of the isilon
> cluster to distribute the traffic / workload for the isilon nodes?
>
> best
> Bjørn

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Hi Skylar,

Skylar Thompson wrote:
> One thing to be aware of with partial incremental backups is the danger of
> backing up data multiple times if the mount points are nested. For
> instance,
>
> /mnt/backup/some-dir
> /mnt/backup/some-dir/another-dir
>
> Under normal operation, a node with DOMAIN set to "/mnt/backup/some-dir
> /mnt/backup/some-dir/another-dir" will backup the contents of /mnt/backup/some-dir/another-dir
> as a separate filespace, *and also* will backup another-dir as a
> subdirectory of the /mnt/backup/some-dir filespace. We reported this as a
> bug, and IBM pointed us at this flag that can be passed as a scheduler
> option to prevent this:
>
> -TESTFLAG=VMPUNDERNFSENABLED

good point,
even if my script works a little bit differently:
by now the starting folder is not red from the "dsm.opt" file but given
in the configuration file for my script "dsmci.cfg". so one run can work
for one node starting on a subfolder (done si as windows has no
VIRTUALMOUNTPOINT option)
Within the this config file several starting folders can be declared and
my script creates in the first step a global list of all folders to be
backed up "partially incremental"

=> well, i'm not sure if i check for multiple entries in the list
=> and if the nesting is done on a deeper level than the list is created
from, i think i won't be aware of such a set-up

i will check this -- thanks for the advice!

best
Bjørn

> On Tue, Jul 17, 2018 at 04:12:17PM +0200, Bjrn Nachtwey wrote:
>> Hi Zoltan,
>>
>> OK, i will translate my text as there are some more approaches discussed :-)
>>
>> breaking up the filesystems in several nodes will work as long as the nodes
>> are of suffiecient size.
>>
>> I'm not sure if a PROXY node will solve the problem, because each "member
>> node" will backup the whole mountpoint. You will need to do partial
>> incremental backups. I expect you will do this based on folders, do you?
>> So, some questions:
>> 1) how will you distribute the folders to the nodes?
>> 2) how will you ensure new folders are processed by one of your "member
>> nodes"? On our filers many folders are created and deleted, sometimes a
>> whole bunch every day. So for me, it was no option to maintain the option
>> file manually. The approach from my script / "MAGS" does this somehow
>> "automatically".
>> 3) what happens if the folders grew not evenly and all the big ones are
>> backed up by one of your nodes? (OK you can change the distribution or even
>> add another node)
>> 4) Are you going to map each backupnode to different nodes of the isilon
>> cluster to distribute the traffic / workload for the isilon nodes?
>>
>> best
>> Bjørn
>

--
--------------------------------------------------------------------------------------------------
Bjørn Nachtwey

Arbeitsgruppe "IT-Infrastruktur“
Tel.: +49 551 201-2181, E-Mail: bjoern.nachtwey@gwdg.de
--------------------------------------------------------------------------------------------------
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de
Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: gwdg@gwdg.de
Service-Hotline: Tel.: +49 551 201-1523, E-Mail: support@gwdg.de
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lossau
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen, Handelsregister-Nr. B 598
--------------------------------------------------------------------------------------------------
Zertifiziert nach ISO 9001
--------------------------------------------------------------------------------------------------
This message was imported via the External PhorumMail Module
@All

possibly the biggest issue when backing up massive file systems in parallel with multiple dsmc processes is expiration. Once you back up a directory with “subdir no”, a no longer existing directory object on that level is expired properly and becomes inactive. However everything underneath that remains active and doesn’t expire (ever) unless you run a “full” incremental on the level above (with “subdir yes”) - and that kind of defeats the purpose of parallelisation. Other pitfalls include avoiding swapping, keeping log files consistent (dsmc doesn’t do thread awareness when logging - it assumes being alone), handling the local dedup cache, updating backup timestamps for a file space on the server, distributing load evenly across multiple nodes on a scale-out filer, backing up from snapshots, chunking file systems up into even parts automatically so you don’t end up with lots of small jobs and one big one, dynamically distributing load across multiple “proxies” if one isn’t enough, handling exceptions, handling directories with characters you can’t parse to dsmc via the command line, consolidating results in a single, comprehensible overview similar to the summary of a regular incremental, being able to do it all in reverse for a massively parallel restore… the list is quite long.

We developed MAGS (as mentioned by Del) to cope with all that - and more. I can only recommend trying it out for free.

Regards

Lars Henningsen
General Storage
This message was imported via the External PhorumMail Module
There's a couple ways we've gotten around this problem:

1. For NFS backups, we don't let TSM do partial incremental backups, even
if we have the filesystem split up. Instead, we mount sub-directories of the
filesystem root on our proxy nodes. This has the double advantage of
letting us break up the filesystem into multiple TSM filespaces (giving us
directory-level backup status reporting, and parallelism in TSM when we
have COLLOCG=FILESPACE), and also parallelism at the NFS level when there
are multiple NFS targets we can talk to (as in the case with Isilon).

2. For GPFS backups, in some cases we can setup independent filesets and
let mmbackup process each as a separate filesystem, though we have some
instances where the end users want an entire GPFS filesystem to have one
inode space so they can do atomic moves as renames. In either case,
though, mmbackup does its own "incremental" backups with filelists passed
to "dsmc selective", which don't update the last-backup time on the TSM
filespace. Our workaround has been to run mmbackup via a preschedule
command, and have the actual TSM incremental backup be of an empty
directory (I call them canary directories in our documentation) that's set
as a virtual mountpoint. dsmc will only run the backup portion of its
scheduled task if the preschedule command succeeds, so if mmbackup fails,
the canary never gets backed up, and will raise an alert.

On Wed, Jul 18, 2018 at 03:07:16PM +0200, Lars Henningsen wrote:
> @All
>
> possibly the biggest issue when backing up massive file systems in parallel with multiple dsmc processes is expiration. Once you back up a directory with ???subdir no???, a no longer existing directory object on that level is expired properly and becomes inactive. However everything underneath that remains active and doesn???t expire (ever) unless you run a ???full??? incremental on the level above (with ???subdir yes???) - and that kind of defeats the purpose of parallelisation. Other pitfalls include avoiding swapping, keeping log files consistent (dsmc doesn???t do thread awareness when logging - it assumes being alone), handling the local dedup cache, updating backup timestamps for a file space on the server, distributing load evenly across multiple nodes on a scale-out filer, backing up from snapshots, chunking file systems up into even parts automatically so you don???t end up with lots of small jobs and one big one, dynamically distributing load across multiple ???proxies??? if one isn???t enough, handling exceptions, handling directories with characters you can???t parse to dsmc via the command line, consolidating results in a single, comprehensible overview similar to the summary of a regular incremental, being able to do it all in reverse for a massively parallel restore??? the list is quite long.
>
> We developed MAGS (as mentioned by Del) to cope with all that - and more. I can only recommend trying it out for free.
>
> Regards
>
> Lars Henningsen
> General Storage

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Canary! I like it!
Richard

-----Original Message-----
From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Skylar Thompson
Sent: Thursday, July 19, 2018 10:37 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] Looking for suggestions to deal with large backups not completing in 24-hours

There's a couple ways we've gotten around this problem:

1. For NFS backups, we don't let TSM do partial incremental backups, even if we have the filesystem split up. Instead, we mount sub-directories of the filesystem root on our proxy nodes. This has the double advantage of letting us break up the filesystem into multiple TSM filespaces (giving us directory-level backup status reporting, and parallelism in TSM when we have COLLOCG=FILESPACE), and also parallelism at the NFS level when there are multiple NFS targets we can talk to (as in the case with Isilon).

2. For GPFS backups, in some cases we can setup independent filesets and let mmbackup process each as a separate filesystem, though we have some instances where the end users want an entire GPFS filesystem to have one inode space so they can do atomic moves as renames. In either case, though, mmbackup does its own "incremental" backups with filelists passed to "dsmc selective", which don't update the last-backup time on the TSM filespace. Our workaround has been to run mmbackup via a preschedule command, and have the actual TSM incremental backup be of an empty directory (I call them canary directories in our documentation) that's set as a virtual mountpoint. dsmc will only run the backup portion of its scheduled task if the preschedule command succeeds, so if mmbackup fails, the canary never gets backed up, and will raise an alert.

On Wed, Jul 18, 2018 at 03:07:16PM +0200, Lars Henningsen wrote:
> @All
>
> possibly the biggest issue when backing up massive file systems in parallel with multiple dsmc processes is expiration. Once you back up a directory with ???subdir no???, a no longer existing directory object on that level is expired properly and becomes inactive. However everything underneath that remains active and doesn???t expire (ever) unless you run a ???full??? incremental on the level above (with ???subdir yes???) - and that kind of defeats the purpose of parallelisation. Other pitfalls include avoiding swapping, keeping log files consistent (dsmc doesn???t do thread awareness when logging - it assumes being alone), handling the local dedup cache, updating backup timestamps for a file space on the server, distributing load evenly across multiple nodes on a scale-out filer, backing up from snapshots, chunking file systems up into even parts automatically so you don???t end up with lots of small jobs and one big one, dynamically distributing load across multiple ???proxies??? if one isn???t enough, handling exceptions, handling directories with characters you can???t parse to dsmc via the command line, consolidating results in a single, comprehensible overview similar to the summary of a regular incremental, being able to do it all in reverse for a massively parallel restore??? the list is quite long.
>
> We developed MAGS (as mentioned by Del) to cope with all that - and more. I can only recommend trying it out for free.
>
> Regards
>
> Lars Henningsen
> General Storage

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Is there no journaling/logging service on these Isilions that could be used to maintain a list of changed files and hand-roll a dsmc-selective-with-file-list process similar to what GPFS uses?

Cheers

Steve

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Richard Cowen
Sent: Friday, 20 July 2018 6:15 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] Looking for suggestions to deal with large backups not completing in 24-hours

Canary! I like it!
Richard

-----Original Message-----
From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Skylar Thompson
Sent: Thursday, July 19, 2018 10:37 AM
To: ADSM-L@VM.MARIST.EDU
Subject: Re: [ADSM-L] Looking for suggestions to deal with large backups not completing in 24-hours

There's a couple ways we've gotten around this problem:

1. For NFS backups, we don't let TSM do partial incremental backups, even if we have the filesystem split up. Instead, we mount sub-directories of the filesystem root on our proxy nodes. This has the double advantage of letting us break up the filesystem into multiple TSM filespaces (giving us directory-level backup status reporting, and parallelism in TSM when we have COLLOCG=FILESPACE), and also parallelism at the NFS level when there are multiple NFS targets we can talk to (as in the case with Isilon).

2. For GPFS backups, in some cases we can setup independent filesets and let mmbackup process each as a separate filesystem, though we have some instances where the end users want an entire GPFS filesystem to have one inode space so they can do atomic moves as renames. In either case, though, mmbackup does its own "incremental" backups with filelists passed to "dsmc selective", which don't update the last-backup time on the TSM filespace. Our workaround has been to run mmbackup via a preschedule command, and have the actual TSM incremental backup be of an empty directory (I call them canary directories in our documentation) that's set as a virtual mountpoint.. dsmc will only run the backup portion of its scheduled task if the preschedule command succeeds, so if mmbackup fails, the canary never gets backed up, and will raise an alert.

On Wed, Jul 18, 2018 at 03:07:16PM +0200, Lars Henningsen wrote:
> @All
>
> possibly the biggest issue when backing up massive file systems in parallel with multiple dsmc processes is expiration. Once you back up a directory with ???subdir no???, a no longer existing directory object on that level is expired properly and becomes inactive. However everything underneath that remains active and doesn???t expire (ever) unless you run a ???full??? incremental on the level above (with ???subdir yes???) - and that kind of defeats the purpose of parallelisation. Other pitfalls include avoiding swapping, keeping log files consistent (dsmc doesn???t do thread awareness when logging - it assumes being alone), handling the local dedup cache, updating backup timestamps for a file space on the server, distributing load evenly across multiple nodes on a scale-out filer, backing up from snapshots, chunking file systems up into even parts automatically so you don???t end up with lots of small jobs and one big one, dynamically distributing load across multiple ???proxies??? if one isn???t enough, handling exceptions, handling directories with characters you can???t parse to dsmc via the command line, consolidating results in a single, comprehensible overview similar to the summary of a regular incremental, being able to do it all in reverse for a massively parallel restore??? the list is quite long.
>
> We developed MAGS (as mentioned by Del) to cope with all that - and more.. I can only recommend trying it out for free.
>
> Regards
>
> Lars Henningsen
> General Storage

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine

This message and any attachment is confidential and may be privileged or otherwise protected from disclosure. You should immediately delete the message if you are not the intended recipient. If you have received this email by mistake please delete it from your system; you should not copy the message or disclose its content to anyone.

This electronic communication may contain general financial product advice but should not be relied upon or construed as a recommendation of any financial product. The information has been prepared without taking into account your objectives, financial situation or needs. You should consider the Product Disclosure Statement relating to the financial product and consult your financial adviser before making a decision about whether to acquire, hold or dispose of a financial product.

For further details on the financial product please go to http://www.bt.com.au

Past performance is not a reliable indicator of future performance.
This message was imported via the External PhorumMail Module
Sadly, no. I made a feature request for this years ago (back when Isilon
was Isilon) but it didn't go anywhere. At this point, our days of running
Isilon storage are numbered, and we'll be investing in DDN/GPFS for the
forseeable future, so I haven't really had leverage to push Dell/EMC/Isilon
on the matter.

On Thu, Jul 19, 2018 at 11:31:06PM +0000, Harris, Steven wrote:
> Is there no journaling/logging service on these Isilions that could be used to maintain a list of changed files and hand-roll a dsmc-selective-with-file-list process similar to what GPFS uses?
>
> Cheers
>
> Steve
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Richard Cowen
> Sent: Friday, 20 July 2018 6:15 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] Looking for suggestions to deal with large backups not completing in 24-hours
>
> Canary! I like it!
> Richard
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager <ADSM-L@VM.MARIST.EDU> On Behalf Of Skylar Thompson
> Sent: Thursday, July 19, 2018 10:37 AM
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: [ADSM-L] Looking for suggestions to deal with large backups not completing in 24-hours
>
> There's a couple ways we've gotten around this problem:
>
> 1. For NFS backups, we don't let TSM do partial incremental backups, even if we have the filesystem split up. Instead, we mount sub-directories of the filesystem root on our proxy nodes. This has the double advantage of letting us break up the filesystem into multiple TSM filespaces (giving us directory-level backup status reporting, and parallelism in TSM when we have COLLOCG=FILESPACE), and also parallelism at the NFS level when there are multiple NFS targets we can talk to (as in the case with Isilon).
>
> 2. For GPFS backups, in some cases we can setup independent filesets and let mmbackup process each as a separate filesystem, though we have some instances where the end users want an entire GPFS filesystem to have one inode space so they can do atomic moves as renames. In either case, though, mmbackup does its own "incremental" backups with filelists passed to "dsmc selective", which don't update the last-backup time on the TSM filespace. Our workaround has been to run mmbackup via a preschedule command, and have the actual TSM incremental backup be of an empty directory (I call them canary directories in our documentation) that's set as a virtual mountpoint. dsmc will only run the backup portion of its scheduled task if the preschedule command succeeds, so if mmbackup fails, the canary never gets backed up, and will raise an alert.
>
> On Wed, Jul 18, 2018 at 03:07:16PM +0200, Lars Henningsen wrote:
> > @All
> >
> > possibly the biggest issue when backing up massive file systems in parallel with multiple dsmc processes is expiration. Once you back up a directory with ???subdir no???, a no longer existing directory object on that level is expired properly and becomes inactive. However everything underneath that remains active and doesn???t expire (ever) unless you run a ???full??? incremental on the level above (with ???subdir yes???) - and that kind of defeats the purpose of parallelisation. Other pitfalls include avoiding swapping, keeping log files consistent (dsmc doesn???t do thread awareness when logging - it assumes being alone), handling the local dedup cache, updating backup timestamps for a file space on the server, distributing load evenly across multiple nodes on a scale-out filer, backing up from snapshots, chunking file systems up into even parts automatically so you don???t end up with lots of small jobs and one big one, dynamically distributing load across multiple ???proxies??? if one isn???t enough, handling exceptions, handling directories with characters you can???t parse to dsmc via the command line, consolidating results in a single, comprehensible overview similar to the summary of a regular incremental, being able to do it all in reverse for a massively parallel restore??? the list is quite long.
> >
> > We developed MAGS (as mentioned by Del) to cope with all that - and more. I can only recommend trying it out for free.
> >
> > Regards
> >
> > Lars Henningsen
> > General Storage
>
> --
> -- Skylar Thompson (skylar2@u.washington.edu)
> -- Genome Sciences Department, System Administrator
> -- Foege Building S046, (206)-685-7354
> -- University of Washington School of Medicine
>
> This message and any attachment is confidential and may be privileged or otherwise protected from disclosure. You should immediately delete the message if you are not the intended recipient. If you have received this email by mistake please delete it from your system; you should not copy the message or disclose its content to anyone.
>
> This electronic communication may contain general financial product advice but should not be relied upon or construed as a recommendation of any financial product. The information has been prepared without taking into account your objectives, financial situation or needs. You should consider the Product Disclosure Statement relating to the financial product and consult your financial adviser before making a decision about whether to acquire, hold or dispose of a financial product.
>
> For further details on the financial product please go to http://www.bt.com.au
>
> Past performance is not a reliable indicator of future performance.

--
-- Skylar Thompson (skylar2@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
This message was imported via the External PhorumMail Module
Hi all,

yes there's a special daemon that might be used -- in theory :-)
in pratice it worked only for small filesystem sizes ... and if it's
filled partially.

A guy from the concat company did some tests and told me they were
totally disappointing as this deamon consumes too many ressources if you
let it write a protocol file which you can use to identify changed,
added and deleted files.
But as far as i know it didn't give a list of changed files just logs
all changes on the files with the kind of the change.

@Lars (Henningsen): Do you know some more details?

best
Bjørn

Skylar Thompson wrote:
> Sadly, no. I made a feature request for this years ago (back when Isilon
> was Isilon) but it didn't go anywhere. At this point, our days of running
> Isilon storage are numbered, and we'll be investing in DDN/GPFS for the
> forseeable future, so I haven't really had leverage to push Dell/EMC/Isilon
> on the matter.
>
> On Thu, Jul 19, 2018 at 11:31:06PM +0000, Harris, Steven wrote:
>> Is there no journaling/logging service on these Isilions that could be used to maintain a list of changed files and hand-roll a dsmc-selective-with-file-list process similar to what GPFS uses?
>>
>> Cheers
>>
>> Steve

>> [...]
This message was imported via the External PhorumMail Module
Hi Bjørn,

actually they improved the isi change list a lot with OneFS8 and performance is no longer really that much of an issue - at least not with 8 to 9-figure number of objects in the file system. My problem (and the main reason why we haven’t integrated it yet into MAGS) is that it is a 99.9% kind of a function. Just like NetApp Snapdiff, you’d always have to recommend doing periodic full incrementals to catch whatever was missed when calculating the snapshot difference (and there usually is). Just like with Snapdiff, you’ll have the back and forth when it comes to who’s problem it is if it doesn’t work and what combination of client and OS on the filer is supported. At the end of the day you’ll have to be able to run a full incremental within an acceptable period of time - and if you have to be able to do that anyway, why not make backup fast enough to run the real thing every day? And using a journal or snapshot difference for backup doesn’t benefit restores one bit, of course.

Regards

Lars Henningsen
General Storage


> On 20. Jul 2018, at 15:48, Bjørn Nachtwey <bjoern.nachtwey@GWDG.DE> wrote:
>
> Hi all,
>
> yes there's a special daemon that might be used -- in theory :-)
> in pratice it worked only for small filesystem sizes ... and if it's filled partially.
>
> A guy from the concat company did some tests and told me they were totally disappointing as this deamon consumes too many ressources if you let it write a protocol file which you can use to identify changed, added and deleted files.
> But as far as i know it didn't give a list of changed files just logs all changes on the files with the kind of the change.
>
> @Lars (Henningsen): Do you know some more details?
>
> best
> Bjørn
>
> Skylar Thompson wrote:
>> Sadly, no. I made a feature request for this years ago (back when Isilon
>> was Isilon) but it didn't go anywhere. At this point, our days of running
>> Isilon storage are numbered, and we'll be investing in DDN/GPFS for the
>> forseeable future, so I haven't really had leverage to push Dell/EMC/Isilon
>> on the matter.
>> On Thu, Jul 19, 2018 at 11:31:06PM +0000, Harris, Steven wrote:
>>> Is there no journaling/logging service on these Isilions that could be used to maintain a list of changed files and hand-roll a dsmc-selective-with-file-list process similar to what GPFS uses?
>>>
>>> Cheers
>>>
>>> Steve
>
> >> [...]
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login