Tim Mooney wrote:
In regard to: Re: [Networker] Multiple drives for recovery?, George...:
Yes, we've seen NetWorker parallelize multi-volume recovers. Most of
the
time it works pretty well. IIRC, this is something that was added in
the
7.x series (earlier versions would always serialize volume access). It
used to be configurable by creating a file in /nsr/debug (do a substring
search of the mailing list archives for striped_recover for more info).
I find it hard to believe that NW can utilize multiple drives. How
does it merge and/or munge everything properly? What if you're instead
recovering from multiple fulls?
Huh? Recovering from multiple fulls? When (and why?!) would you recover
from multiple fulls, simultaneously, and restore all of the data into the
original location? I have to be misunderstanding something, cause I'm not
following.
What I was speaking to was the case wherein the saveset spans, lets say
three tapes, for example, if NW could recover all the data from
volume1of3 while it's also recovering the data from 2of3 and 3of3
simultaneously, etc. and thereby save time versus reading them one at a
time. It would somehow have to merge them together on disk once done.
That was my understanding of parallel recover, but looks like I was
reaching beyond what the term implies.
Think about the browse process. Let's say you need to recover three files
from a directory. fileA never changes, so it only gets backed up when you
do a full backup. fileB changes once a week, so it only gets backed up on
your Saturday backups. fileC changes daily, so it gets backed up every
day.
Now you browse the index, find that directory, do a
add fileA fileB fileC
and then run
volumes
and it shows you that it will require 3 tape volumes.
NetWorker knows that only one copy of fileA is coming back (from the
full), only one copy of fileB is coming back (from the tape from Saturday)
and only one copy of fileC is coming back (from the most recently daily).
It can parallelize the recovery because there are no conflicts.
Now I anticipate it would make the coding much simpler if NetWorker
always recovered from backups of a lower level first, so your full
would be the first one that got recovered. If fileB and fileC were
both backed up at level 'incr', there's no reason why they couldn't be
recovered in either order or completely in parallel.
Yes, I follow you.
How can it temporarily store all that
data as disk space could become jeopardized at some point. How does it
organize and/or re-conglomerate all that later?
I'm totally not following.
We have, however, seen a few instances where recover apparently
deadlocks
in the striped recovery code. This happened to us to a couple of times
under 7.2.x or 7.4.x, but we upgraded to 7.5.2 last week and the
first big
recover we had to do triggered a deadlock in recovery. We've had a case
open with EMC about this issue since last Friday.
What do you mean by 'deadlocks'?
I mean pauses indefinitely and can't seem to be "prodded" into continuing.
The recovery process stops after one or more tapes have been read (so
it's part way through the process of recovering the files that were
requested) and never proceeds with subsequent tapes.
There are reasons why this can happen, like media database corruption
(Networker knows the ssids it needs but can't figure out which tapes
they're on) or issues with the jukebox resource (it's partially corrupt
and NetWorker knows the tape is nearline but doesn't know where it is),
but both of those issues were ruled out in the case we had open.
I was thinking to try upgrading the client software on client A, but I
doubt that has any effect over what the server decides to do on its
end in terms of loading those tapes.
That's exactly what I thought, but it turns out to be incorrect, much to
my great surprise. I had assumed that other than its role in selecting
the files to be recovered, the client couldn't possibly have any influence
on how the server goes about the process of actually finding the data and
feeding it back to the client, but it looks like there's more going on
there than I understand.
We got a resolution from EMC this morning that was able to resolve the
deadlock issue we were seeing. Upgrading the client software from 7.4.2
to 7.5.2 (to match the server) fixed the issue. I still don't understand
how the client could be influencing this, but it appears it was.
I'll try that. I did create the /nsr/debug/no_striped_recover file. That
did the job as far as forcing NW to sequentially load the tapes, BUT
after loading the full first, and recovering the data there, it then
loads the incr tapes one at a time, but in the seemingly wrong order,
going from newest to oldest rather than the converse as I would have
expected!!! After the recover completed, I ran my validation, and this
time there were only 3 bad matches which were the mod times for three
recovered directories as opposed to hundreds of directories as before.
Otherwise, no differences were found. This is clearly much closer than
before, but still 3 mod times away from being perfect. Maybe the client
upgrade will do the trick.
George
Tim
--
George Sinclair
Voice: (301) 713-3284 x210
- The preceding message is personal and does not reflect any official or
unofficial position of the United States Department of Commerce -
- Any opinions expressed in this message are NOT those of the US Govt. -
via RSS at
http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
