Backup Central

I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

Jon
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

On Monday, November 01, 2010 10:28:24 pm Jon LaBadie did opine:

I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

Jon

I just checked my latest man page for amanda.conf that goes with 3.2.0, and
its not there. I presume it has been deprecated. NDI when Jon.

My scripts do something similar, except they append them to the just used
vtape. I think, there has been some tom foolery with the data symlink
recently, so let me check.

Yup, its all working:

#> ls -l /amandatapes/Dailys/data/
total 14261572
-rw------- 1 amanda amanda 32768 2010-11-01 01:00 00000.Dailys-5
-rw------- 1 amanda amanda 16864519 2010-11-01 01:00 00001.shop._home.1
-rw------- 1 amanda amanda 12113932288 2010-11-01 01:30 00002.coyote._usr_movies.0
-rw------- 1 amanda amanda 1695668442 2010-11-01 01:31 00003.coyote._home.2
-rw------- 1 amanda amanda 341323992 2010-11-01 01:31 00004.coyote._tmp.1
-rw------- 1 amanda amanda 97668334 2010-11-01 01:31 00005.coyote._var.1
-rw------- 1 amanda amanda 90500843 2010-11-01 01:31 00006.coyote._GenesAmandaHelper-0.6.3
-rw------- 1 amanda amanda 32533484 2010-11-01 01:31 00007.coyote._usr_lib.2
-rw------- 1 amanda amanda 13150208 2010-11-01 01:31 00008.coyote._usr_bin.1
-rw------- 1 amanda amanda 7896190 2010-11-01 01:31 00009.coyote._usr_local.1
-rw------- 1 amanda amanda 3440052 2010-11-01 01:31 00010.coyote._usr_src.1
-rw------- 1 amanda amanda 827460 2010-11-01 01:31 00011.coyote._usr_share.2
-rw------- 1 amanda amanda 459477 2010-11-01 01:31 00012.coyote._lib.1
-rw------- 1 amanda amanda 444168 2010-11-01 01:31 00013.coyote._opt.1
-rw------- 1 amanda amanda 186368 2010-11-01 01:31 00014.coyote._usr_pix.1
-rw------- 1 amanda amanda 155883 2010-11-01 01:31 00015.shop._usr_src.1
-rw------- 1 amanda amanda 150919 2010-11-01 01:31 00016.shop._var_lib_amanda.1
-rw------- 1 amanda amanda 118275 2010-11-01 01:31 00017.coyote._etc.1
-rw------- 1 amanda amanda 73728 2010-11-01 01:31 00018.coyote._usr_dlds_misc.1
-rw------- 1 amanda amanda 62141 2010-11-01 01:31 00019.coyote._usr_include.1
-rw------- 1 amanda amanda 53295 2010-11-01 01:31 00020.shop._etc.1
-rw------- 1 amanda amanda 49437 2010-11-01 01:31 00021.coyote._root.1
-rw------- 1 amanda amanda 43008 2010-11-01 01:31 00022.coyote._usr_music.1
-rw------- 1 amanda amanda 43008 2010-11-01 01:31 00023.coyote._usr_dlds_tgzs.1
-rw------- 1 amanda amanda 43008 2010-11-01 01:31 00024.coyote._usr_dlds_rpms.1
-rw------- 1 amanda amanda 43008 2010-11-01 01:31 00025.coyote._sbin.1
-rw------- 1 amanda amanda 43008 2010-11-01 01:31 00026.coyote._boot.1
-rw------- 1 amanda amanda 43008 2010-11-01 01:31 00027.coyote._bin.1
-rw------- 1 amanda amanda 35309 2010-11-01 01:31 00028.coyote._usr_sbin.1
-rw------- 1 amanda amanda 33965 2010-11-01 01:31 00029.shop._usr_lib_amanda.1
-rw------- 1 amanda amanda 33604 2010-11-01 01:31 00030.shop._usr_local.1
-rw------- 1 amanda amanda 32966 2010-11-01 01:31 00031.coyote._usr_X11R6.1
-rw------- 1 amanda amanda 32981 2010-11-01 01:31 00032.coyote._usr_libexec.1
-rw-r--r-- 1 amanda amanda 112640 2010-11-01 01:32 configuration.tar
-rw-r--r-- 1 amanda amanda 173271040 2010-11-01 01:32 indices.tar

The last two files are my config dir, and all the indices, as of the end of
that amdump run this morning.

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Bing's Rule:
Don't try to stem the tide -- move the beach.

On Mon, Nov 01, 2010 at 10:39:06PM -0400, Gene Heskett wrote:
On Monday, November 01, 2010 10:28:24 pm Jon LaBadie did opine:

I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

Jon

I just checked my latest man page for amanda.conf that goes with 3.2.0, and
its not there. I presume it has been deprecated. NDI when Jon.

I'm running 3.2.0 also. Both my manpage and that on the wiki have autoflush.

New glasses?

Jon
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

On Tuesday, November 02, 2010 12:08:58 am Jon LaBadie did opine:

On Mon, Nov 01, 2010 at 10:39:06PM -0400, Gene Heskett wrote:
On Monday, November 01, 2010 10:28:24 pm Jon LaBadie did opine:
I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

Jon

I just checked my latest man page for amanda.conf that goes with
3.2.0, and its not there. I presume it has been deprecated. NDI when
Jon.

I'm running 3.2.0 also. Both my manpage and that on the wiki have
autoflush.

New glasses?

I don't think so Jon. My man pages get updated with every build
& install, so they should be current. Yes, this files date is2010-10-29 14:16
so it was installed when I last built 3.20 with my scripts.

From that file - /usr/local/share/man/man5/amanda.conf:
taperflush int
Default: 0. At the end of a run, Amanda will start a new tape to flush remaining data if there is more data on the holding
disk at the end of a run than this setting allows; the amount is specified as a percentage of the capacity of a single
volume. In other words, at the end of a run, Amanda will begin a new tape if the inequality h > t × f is satisfied, where
h is the amount of data remaining on the holding disk from this or previous runs, t is the capacity of a volume, and f is
this parameter, expressed as a percentage. This parameter may be greater than 100%.

The value of this parameter may not exceed that of the flush-threshold-scheduled parameter.; autoflush must be set to
´yes´ if taperflush is greater than 0.

And that is the only place in the whole manpage it is mentioned.

From the way I'd read that it seems to have been replaced by taperflush
which in turn has some fawncy math to trigger it. This time of the night,
I can't grok what it really means as I took my bedtime pills 30 minutes ago.

Maybe you can decipher it and explain it to me?

Jon


--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The full potentialities of human fury cannot be reached until a friend
of both parties tactfully interferes.
-- G.K. Chesterton

On 11/1/10 10:10 PM, Jon LaBadie wrote:
I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

How do you achieve the separate dump? Is it the same configuration with command line options to
stipulate what is getting done? It doesn't make sense that something as fundamental as autoflush
would stop working or be removed from the options, so I'm just wondering how you have it set up.
Details?


--
---------------

Chris Hoogendyk

-
O__ ---- Systems Administrator
c/ /'_ --- Biology& Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst

<hoogendyk < at > bio.umass.edu>

---------------

Erdös 4

On Tuesday, November 02, 2010 09:11:44 am Chris Hoogendyk did opine:

On 11/1/10 10:10 PM, Jon LaBadie wrote:
I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

How do you achieve the separate dump? Is it the same configuration with
command line options to stipulate what is getting done? It doesn't make
sense that something as fundamental as autoflush would stop working or
be removed from the options, so I'm just wondering how you have it set
up. Details?

I agree, Chris. And the keyword autoflush doesn't grep in the ChangeLog,
Hence we have had no warning of a change in how it works. I have
previously commented on the fact that such behavior changes BELONG
in the ChangeLog!

From the latest 3.2.0-svn-3583 man page for amanda.conf:
(long lines warning)

taperflush int
Default: 0. At the end of a run, Amanda will start a new tape to flush remaining data if there is more data on the holding
disk at the end of a run than this setting allows; the amount is specified as a percentage of the capacity of a single
volume. In other words, at the end of a run, Amanda will begin a new tape if the inequality h > t × f is satisfied, where
h is the amount of data remaining on the holding disk from this or previous runs, t is the capacity of a volume, and f is
this parameter, expressed as a percentage. This parameter may be greater than 100%.

The value of this parameter may not exceed that of the flush-threshold-scheduled parameter.; autoflush must be set to
´yes´ if taperflush is greater than 0.

This is now the only place in the man page where it is mentioned. I presume
one could munch his way through that logic and arrive at similar results to
the former Just Works(TM) but I fail to see how this can possibly be an
improvement. IMO "autoflush yes" ought to be an absolute command,
where it does that first, and then proceeds to do the rest of the backup
session using the taperalgo selected with prior knowledge of how much tape
has already been used.

This, if that is how amanda now works, is a huge regression IMO.

--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Forest fires cause Smokey Bears.

On Tue, Nov 02, 2010 at 07:26:47AM -0400, Chris Hoogendyk wrote:


On 11/1/10 10:10 PM, Jon LaBadie wrote:
I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

How do you achieve the separate dump? Is it the same configuration
with command line options to stipulate what is getting done? It
doesn't make sense that something as fundamental as autoflush would
stop working or be removed from the options, so I'm just wondering
how you have it set up. Details?


My scheme would not scale well or be suitable in a
production environment.

There is only one config, "Daily". For the amanda
server, one DLE (called Amanda) uses a disk of "/"
and "includes" each of the amanda directories.
My other server DLEs exclude these same dirs.

My crontab executes a script that does two amdumps.
The first looks like this:

/usr/sbin/amdump Daily \
.butch. Root \
.butch. Var \
<snip bunch of similar DLE lines>
.bigcow. Opt \
.bigcow. Home

Then it executes:

/usr/sbin/amdump Daily --no-taper .bigcow. Amanda

to get the amanda files.

--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

Jon LaBadie wrote:
On Tue, Nov 02, 2010 at 07:26:47AM -0400, Chris Hoogendyk wrote:

On 11/1/10 10:10 PM, Jon LaBadie wrote:

I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?

How do you achieve the separate dump? Is it the same configuration
with command line options to stipulate what is getting done? It
doesn't make sense that something as fundamental as autoflush would
stop working or be removed from the options, so I'm just wondering
how you have it set up. Details?



My scheme would not scale well or be suitable in a
production environment.

There is only one config, "Daily". For the amanda
server, one DLE (called Amanda) uses a disk of "/"
and "includes" each of the amanda directories.
My other server DLEs exclude these same dirs.

My crontab executes a script that does two amdumps.
The first looks like this:

/usr/sbin/amdump Daily \
.butch. Root \
.butch. Var \
<snip bunch of similar DLE lines>
.bigcow. Opt \
.bigcow. Home

Amanda dump and flush only the listed dle
It is not flushing '.bigcow. Amanda' because it is not on the command line.

We can change this behavior.

What user expect from dle on the amdump command line:
- dump/flush only the listed dle (actual behavior)
or
- dump only the listed dle and flush everything?

Jean-Louis

On 11/2/10 10:59 AM, Jean-Louis Martineau wrote:
Jon LaBadie wrote:
On Tue, Nov 02, 2010 at 07:26:47AM -0400, Chris Hoogendyk wrote:
On 11/1/10 10:10 PM, Jon LaBadie wrote:
I've started backing up my amanda files nightly
in a separate amdump run after my primary amdump.
My plan was to leave it on the holding disk and
let it autoflush to vtape during the next day's
dump. Unfortunately it does not flush. I've
now got about 7 dumps on the holding disk.

Any suggestions why an "autoflush yes" might
not be effective?
How do you achieve the separate dump? Is it the same configuration
with command line options to stipulate what is getting done? It
doesn't make sense that something as fundamental as autoflush would
stop working or be removed from the options, so I'm just wondering
how you have it set up. Details?


My scheme would not scale well or be suitable in a
production environment.

There is only one config, "Daily". For the amanda
server, one DLE (called Amanda) uses a disk of "/"
and "includes" each of the amanda directories.
My other server DLEs exclude these same dirs.

My crontab executes a script that does two amdumps.
The first looks like this:

/usr/sbin/amdump Daily \
.butch. Root \
.butch. Var \
<snip bunch of similar DLE lines>
.bigcow. Opt \
.bigcow. Home
Amanda dump and flush only the listed dle
It is not flushing '.bigcow. Amanda' because it is not on the command line.

We can change this behavior.

What user expect from dle on the amdump command line:
- dump/flush only the listed dle (actual behavior)
or
- dump only the listed dle and flush everything?

That depends on how you define everything. I would think that if a dle was neither defined in the
amanda.conf file for the configuration nor listed on the command line, then it should not be
flushed. This would allow for different configurations to share a holding disk.

Otherwise, if it is a dle that is included in the configuration, then it should be flushed if
autoflush is yes.

In my case, my holding disks are larger than is really necessary, so I tend to use the space as
scratch and recovery space fairly often. There can be a lot of cruft there.


--
---------------

Chris Hoogendyk

-
O__ ---- Systems Administrator
c/ /'_ --- Biology& Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst

<hoogendyk < at > bio.umass.edu>

---------------

Erdös 4

Chris Hoogendyk wrote:
Amanda dump and flush only the listed dle
It is not flushing '.bigcow. Amanda' because it is not on the command
line.

We can change this behavior.

What user expect from dle on the amdump command line:
- dump/flush only the listed dle (actual behavior)
or
- dump only the listed dle and flush everything?

That depends on how you define everything. I would think that if a dle
was neither defined in the amanda.conf file for the configuration nor
listed on the command line, then it should not be flushed. This would
allow for different configurations to share a holding disk.

Otherwise, if it is a dle that is included in the configuration, then
it should be flushed if autoflush is yes.

In my case, my holding disks are larger than is really necessary, so I
tend to use the space as scratch and recovery space fairly often.
There can be a lot of cruft there.
Everything is all what would have been flushed if you didn't specify a
dle on the command line, it include dle that are not in the disklist.

You should use separete directory for others configuration or others data.

Jean-Louis

On Tue, Nov 02, 2010 at 10:59:38AM -0400, Jean-Louis Martineau wrote:
Jon LaBadie wrote:

My scheme would not scale well or be suitable in a
production environment.

There is only one config, "Daily". For the amanda
server, one DLE (called Amanda) uses a disk of "/"
and "includes" each of the amanda directories.
My other server DLEs exclude these same dirs.

My crontab executes a script that does two amdumps.
The first looks like this:

/usr/sbin/amdump Daily \
.butch. Root \
.butch. Var \
<snip bunch of similar DLE lines>
.bigcow. Opt \
.bigcow. Home

Amanda dump and flush only the listed dle
It is not flushing '.bigcow. Amanda' because it is not on the command line.

We can change this behavior.

Ahh, if I didn't miss it in the docs, a note should be added.

What user expect from dle on the amdump command line:
- dump/flush only the listed dle (actual behavior)
or
- dump only the listed dle and flush everything?

Certainly I expected any DLE from the specified config
to be flushed on any amdump run. But now that you
describe the behavior I can see either behavior could
be wanted at different times. For example, stuff is
still on the holding disk but I want to make an archive
copy of some DLE(s). Then the current behavior is
desired.

I presume the disklist would be the arbiter for what
things on the holding disk are candidates for flushing.
But might there might also be edge cases where the
disklist was changed and the holding disk still has
dumps from some deleted DLE?

It seems to me that the current behavior would be
suitable in the majority of cases. I don't think
too many amanda installations run a daily amdump
with specified DLEs.

If the current behavior is retained, what would
be the possibility of adding an amdump option
to force flushing even when DLEs are specified?

jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)