Backup Central

I'm glad my tarball worked for you. I'll find out from Jean-Louis how
to generate snapshots with the datestamp, in case I need to do so in
the future.

But you've brought up a new topic, so I've changed the subject line:

On Sun, Feb 28, 2010 at 8:40 AM, Gene Heskett <gene.heskett < at > verizon.net> wrote:
WARNING: shop /home: fallback_splitsize of 10240k < 0.1% of tape size and may
be used for PORT_DUMP; check your configuration

A line for every dle.  Is this something I need to address before tonights
real run?

Yes, I just added this warning. Some refinement may be in order, so
let me know what you think.

The problem is that Amanda's fallback_splitsize defaults to 10M, and
I've seen a number of users recently who are using this splitsize for
multi-terabyte dumps, without knowing it. This generates *terrible*
performance on backup, since the tape drive has to write tens or
hundreds of thousands of filemarks, and also on recovery, since the
recovery app must seek to and read from each of those many thousands
of files.

This is usually due not to a "fallback" in an error condition, but to
misconfiguration. The splitsize parameters are *very* confusing!

So the new warning is intended to alert folks such as yourself to the
potential misconfiguration. It's just a warning, so Amanda will
continue to run as before, but you really should fix it. To fix it,
do one of:
* set split_diskbuffer properly
* raise your fallback_splitsize to something larger than 0.001 * tape-length
* set fallback_splitsize to 0

Also, where do I find the definition of a PORT_DUMP?  That is a new one to
me.

Oops, that should say PORT-WRITE. PORT-WRITE is a dump straight to
the device, without going to holding first. Amanda does PORT-WRITE
when the DLE is larger than the holding disk, or when no holding disk
is configured. It is the opposite of FILE-WRITE, which is a write
from holding disk to the tape device.

Amanda always uses the tape_splitsize for FILE-WRITE dumps. It does
the same for PORT-WRITE, but if split_diskbuffer is not set, which is
the case on your system, then it uses fallback_splitsize instead (with
its default of 10M), with no notice of the "fallback".

So what if I changed the warning to read

WARNING: shop /home: fallback_splitsize of 10240k < 0.1% of tape
length and may create >1000 parts; set split_diskbuffer or increase
fallback_splitsize

I'll also add a wiki page to help explain the background for the error.

Let me know what you think.

Dustin

P.S. There are plans afoot to change these configuration parameters to
something less confusing, but not in the upcoming release.

--
Open Source Storage Engineer
http://www.zmanda.com

On Sun, Feb 28, 2010 at 10:59:23AM -0600, Dustin J. Mitchell wrote:
...

Yes, I just added this warning. Some refinement may be in order, so
let me know what you think.

The problem is that Amanda's fallback_splitsize defaults to 10M, and
I've seen a number of users recently who are using this splitsize for
multi-terabyte dumps, without knowing it. This generates *terrible*
performance on backup, since the tape drive has to write tens or
hundreds of thousands of filemarks, and also on recovery, since the
recovery app must seek to and read from each of those many thousands
of files.

This is usually due not to a "fallback" in an error condition, but to
misconfiguration. The splitsize parameters are *very* confusing!

So the new warning is intended to alert folks such as yourself to the
potential misconfiguration. It's just a warning, so Amanda will
continue to run as before, but you really should fix it. To fix it,
do one of:
* set split_diskbuffer properly
* raise your fallback_splitsize to something larger than 0.001 * tape-length
* set fallback_splitsize to 0

...
Amanda always uses the tape_splitsize for FILE-WRITE dumps. It does
the same for PORT-WRITE, but if split_diskbuffer is not set, which is
the case on your system, then it uses fallback_splitsize instead (with
its default of 10M), with no notice of the "fallback".

So what if I changed the warning to read

WARNING: shop /home: fallback_splitsize of 10240k < 0.1% of tape
length and may create >1000 parts; set split_diskbuffer or increase
fallback_splitsize


Is this a case where verbosity is a positive?


WARNING: shop /home: fallback_splitsize of 10M is < 0.1% of tape length.
This may create > 1000 parts, severely degrading backup/restore performance.
To remedy, create/set a split_diskbuffer or increase fallback_splitsize
See <appropriate_URL> for more information.

jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

On Sun, Feb 28, 2010 at 12:17 PM, Jon LaBadie <jon < at > jgcomp.com> wrote:
Is this a case where verbosity is a positive?


WARNING: shop /home: fallback_splitsize of 10M is < 0.1% of tape length.
This may create > 1000 parts, severely degrading backup/restore performance.
To remedy, create/set a split_diskbuffer or increase fallback_splitsize
See <appropriate_URL> for more information.

Gene mentioned that the error occurs for every one of his DLEs, which
I expect will be the common case. So maybe this verbosity should
occur only once in the amcheck output:

WARNING: shop /home: fallback_splitsize of 10M is < 0.1% of tape length.
This may create > 1000 parts, severely degrading backup/restore performance.
To remedy, create/set a split_diskbuffer or increase fallback_splitsize
See <appropriate_URL> for more information.
WARNING: shop /usr: fallback_splitsize of 10M is < 0.1% of tape length.
WARNING: shop /var: fallback_splitsize of 10M is < 0.1% of tape length.
...

Thanks!

Dustin

--
Open Source Storage Engineer
http://www.zmanda.com

On Sunday 28 February 2010, Dustin J. Mitchell wrote:
I'm glad my tarball worked for you. I'll find out from Jean-Louis how
to generate snapshots with the datestamp, in case I need to do so in
the future.

But you've brought up a new topic, so I've changed the subject line:

On Sun, Feb 28, 2010 at 8:40 AM, Gene Heskett <gene.heskett < at > verizon.net>
wrote:
WARNING: shop /home: fallback_splitsize of 10240k < 0.1% of tape size and
may be used for PORT_DUMP; check your configuration

A line for every dle. Is this something I need to address before
tonights real run?

Yes, I just added this warning. Some refinement may be in order, so
let me know what you think.

The problem is that Amanda's fallback_splitsize defaults to 10M, and
I've seen a number of users recently who are using this splitsize for
multi-terabyte dumps, without knowing it. This generates *terrible*
performance on backup, since the tape drive has to write tens or
hundreds of thousands of filemarks, and also on recovery, since the
recovery app must seek to and read from each of those many thousands
of files.

OhmyGawd. Plumb deadly for tape users.

This is usually due not to a "fallback" in an error condition, but to
misconfiguration. The splitsize parameters are *very* confusing!

I almost understood them, so it can't be too bad, but then it doesn't seem to
be a consideration here.

So the new warning is intended to alert folks such as yourself to the
potential misconfiguration. It's just a warning, so Amanda will
continue to run as before, but you really should fix it. To fix it,
do one of:
* set split_diskbuffer properly
* raise your fallback_splitsize to something larger than 0.001 *
tape-length * set fallback_splitsize to 0

Also, where do I find the definition of a PORT_DUMP? That is a new one
to me.

Oops, that should say PORT-WRITE. PORT-WRITE is a dump straight to
the device, without going to holding first. Amanda does PORT-WRITE
when the DLE is larger than the holding disk, or when no holding disk
is configured. It is the opposite of FILE-WRITE, which is a write
from holding disk to the tape device.

Ahh, clarified, for me at least. As I have holding disk out the yang, and am
using v-tapes anyway, it hadn't stuck up its hand and waved at me before.
Would this not also result in individual split sized files I could see in my
/amanatapes/Dailys partition? In which case it has only happened once that
I'm aware of.

Amanda always uses the tape_splitsize for FILE-WRITE dumps. It does
the same for PORT-WRITE, but if split_diskbuffer is not set, which is
the case on your system, then it uses fallback_splitsize instead (with
its default of 10M), with no notice of the "fallback".

So what if I changed the warning to read

WARNING: shop /home: fallback_splitsize of 10240k < 0.1% of tape
length and may create >1000 parts; set split_diskbuffer or increase
fallback_splitsize

I'd make a note in the example/amanda.conf that this is related to the memory
the machine has, and while it can use swap, this shouldn't normally be more
than say 50% of the memory in the machine. I have 4GB of ram, and 4 to 8 GB
of swap but I don't let it get too far into swap without doing a swapoff -a,
swapon -a to clean up the mess.

This wording makes it more verbose, but certainly worth it for the clarity,
I'd go for it. However, on consideration it may be worthwhile to issue the
warning only when a "df /dump's" free space is less than say 2x the tapesize.

/dumps here is on /, which is a 1TB drive here. Here is mine:

[root < at > coyote dumps]# df /dumps
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 100790036 9563148 86106976 10% /

Not a lot of use bothering the folks if the warning is just another cry of
'wolf'. There really _should_ be a wolf at the door...

I'll also add a wiki page to help explain the background for the error.

Good idea, and I can find the link.

Let me know what you think.

See above.
Thanks, Dustin

P.S. There are plans afoot to change these configuration parameters to
something less confusing, but not in the upcoming release.

I suppose they will bite me when that occurs. NBD, I've been forewarned.

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)

Duty, n:
What one expects from others.
-- Oscar Wilde

On Sunday 28 February 2010, Jon LaBadie wrote:
On Sun, Feb 28, 2010 at 10:59:23AM -0600, Dustin J. Mitchell wrote:
...

Yes, I just added this warning. Some refinement may be in order, so
let me know what you think.

The problem is that Amanda's fallback_splitsize defaults to 10M, and
I've seen a number of users recently who are using this splitsize for
multi-terabyte dumps, without knowing it. This generates *terrible*
performance on backup, since the tape drive has to write tens or
hundreds of thousands of filemarks, and also on recovery, since the
recovery app must seek to and read from each of those many thousands
of files.

This is usually due not to a "fallback" in an error condition, but to
misconfiguration. The splitsize parameters are *very* confusing!

So the new warning is intended to alert folks such as yourself to the
potential misconfiguration. It's just a warning, so Amanda will
continue to run as before, but you really should fix it. To fix it,
do one of:
* set split_diskbuffer properly
* raise your fallback_splitsize to something larger than 0.001 *
tape-length * set fallback_splitsize to 0

...

Amanda always uses the tape_splitsize for FILE-WRITE dumps. It does
the same for PORT-WRITE, but if split_diskbuffer is not set, which is
the case on your system, then it uses fallback_splitsize instead (with
its default of 10M), with no notice of the "fallback".

So what if I changed the warning to read

WARNING: shop /home: fallback_splitsize of 10240k < 0.1% of tape
length and may create >1000 parts; set split_diskbuffer or increase
fallback_splitsize

Is this a case where verbosity is a positive?


WARNING: shop /home: fallback_splitsize of 10M is < 0.1% of tape length.
This may create > 1000 parts, severely degrading backup/restore
performance. To remedy, create/set a split_diskbuffer or increase
fallback_splitsize See <appropriate_URL> for more information.

jl

Yes I think so Jon, but in my case I have over 800GB of /dumps available, so
this particular item somewhat resembles a cry of wolf, when its just a
termite, which I have to call for tomorrow. Its been about 20 years since
we've had it done, and we just discovered some tunnels on the music room
wall, same place as 20 years ago. Persistent critters.

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)

If you're constantly being mistreated, you're cooperating with the treatment.

On Sun, Feb 28, 2010 at 3:04 PM, Gene Heskett <gene.heskett < at > verizon.net> wrote:
Is this a case where verbosity is a positive?


WARNING: shop /home: fallback_splitsize of 10M is < 0.1% of tape length.
This may create > 1000 parts, severely degrading backup/restore
performance. To remedy, create/set a split_diskbuffer or increase
fallback_splitsize See <appropriate_URL> for more information.

jl

Yes I think so Jon, but in my case I have over 800GB of /dumps available, so
this particular item somewhat resembles a cry of wolf, when its just a

I committed a change to make this warning look almost exactly as Jon
described. I also wrote up a nice wiki article on the topic:
http://wiki.zmanda.com/index.php/Splitsize_too_small

Please edit it up if you see room for improvements.

Dustin

--
Open Source Storage Engineer
http://www.zmanda.com

On Monday 01 March 2010, Dustin J. Mitchell wrote:
On Sun, Feb 28, 2010 at 3:04 PM, Gene Heskett <gene.heskett < at > verizon.net>
wrote:
Is this a case where verbosity is a positive?


WARNING: shop /home: fallback_splitsize of 10M is < 0.1% of tape length.
This may create > 1000 parts, severely degrading backup/restore
performance. To remedy, create/set a split_diskbuffer or increase
fallback_splitsize See <appropriate_URL> for more information.

jl

Yes I think so Jon, but in my case I have over 800GB of /dumps available,
so this particular item somewhat resembles a cry of wolf, when its just a

I committed a change to make this warning look almost exactly as Jon
described. I also wrote up a nice wiki article on the topic:
http://wiki.zmanda.com/index.php/Splitsize_too_small

Please edit it up if you see room for improvements.

Dustin

Looks good to me, Dustin. In my case, I had converted all local DLE's to
using dt_amgtar, but didn't realize that this format also required all the
stuff that was included in the global definition for completeness. Since the
'out on the network' DLE's are still using the global definitions, I
corrected both, and now amcheck seems to be happy.

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)

Option Paralysis:
The tendency, when given unlimited choices, to make none.
-- Douglas Coupland, "Generation X: Tales for an Accelerated
Culture"