Backup Central

I have Amanda configured to do all its backups (local and net) to disk.
I set the "reserved" parameter to 0 -- that will make it do complete
backups, right? (As opposed to degraded mode)

And if I run amflush I'll have complete backups on tape of everything in
the disklist, right?

<long story>

A while back, I bought a Quantum VS160 DLT tape drive and found the
amanda backup configuration I'd been using for years (to DDS) wouldn't
stream it.

After much email traffic and hardware futzing, I've come to the
conclusion that it is not possible for my system (linux 2.6, 2.8GHx P4,
2GB RAM) to stream data to that tape drive from one part of a disk (IDE
or SCSI) while writing to another part. But just reading from a
contiguous file on a holdingdisk that isn't doing *anything* else works
fine.

I think the bare disk drives may be fast enough, but by the time the
drivers and OS get through buffering data and moving it around, the tape
has emptied its I/O buffer and stopped.

The seemingly working configuration:

The holdingdisk is a dedicated 7200 rpm 120GB IDE on its own controller
in a PCI slot. DMA on, drive is a master, 80 conductor cable.

Amanda's configuration puts all the filesystems on a single spindle on
separate spindles and uses gtar as the archiving program (Linux 2.6's
dump kept reporting errors -- maybe because the server is gentoo, and
the clients are RH and debian).

Amdump runs every night, and the reports look reasonable.

Every once in a while, a shell script runs that checks to see how much
data is waiting on the holdingdisk. If it's a significant fraction of
the tape amflush moves the backups to tape, and amverifyrun verifies the
tape.

This seems to be working. Amflush streams the tape (except between tape
files); the tapes verify; files are on the tape; and I can get to them
with amrecover. But just to be sure, do you people think this is a
reasonable solution?

</long story>

Thanks to everyone for your help. This has been quite an education.

--
Glenn English <ghe < at > slsware.com>

Glenn English wrote:
I have Amanda configured to do all its backups (local and net) to disk.
I set the "reserved" parameter to 0 -- that will make it do complete
backups, right? (As opposed to degraded mode)

And if I run amflush I'll have complete backups on tape of everything in
the disklist, right?

<long story>

A while back, I bought a Quantum VS160 DLT tape drive and found the
amanda backup configuration I'd been using for years (to DDS) wouldn't
stream it.

After much email traffic and hardware futzing, I've come to the
conclusion that it is not possible for my system (linux 2.6, 2.8GHx P4,
2GB RAM) to stream data to that tape drive from one part of a disk (IDE
or SCSI) while writing to another part. But just reading from a
contiguous file on a holdingdisk that isn't doing *anything* else works
fine.

I think the bare disk drives may be fast enough, but by the time the
drivers and OS get through buffering data and moving it around, the tape
has emptied its I/O buffer and stopped.

The seemingly working configuration:

The holdingdisk is a dedicated 7200 rpm 120GB IDE on its own controller
in a PCI slot. DMA on, drive is a master, 80 conductor cable.

Amanda's configuration puts all the filesystems on a single spindle on
separate spindles and uses gtar as the archiving program (Linux 2.6's
dump kept reporting errors -- maybe because the server is gentoo, and
the clients are RH and debian).

Amdump runs every night, and the reports look reasonable.

Every once in a while, a shell script runs that checks to see how much
data is waiting on the holdingdisk. If it's a significant fraction of
the tape amflush moves the backups to tape, and amverifyrun verifies the
tape.

This seems to be working. Amflush streams the tape (except between tape
files); the tapes verify; files are on the tape; and I can get to them
with amrecover. But just to be sure, do you people think this is a
reasonable solution?

</long story>

Makes complete sense to me.

I don't know how fast a VS160 DLT tapedrive is, but given that you
have plenty of RAM, you could try a normal run once again with
a large value for "tapebufs" in amanda.conf, like 1000 or more,
resulting in a 32 Mbyte buffer (the default is 20 bufs * 32Kbyte).


--
Paul Bijnens, Xplanation Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens < at > xplanation.com
***********************************************************************
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***********************************************************************

On Wed, Jun 16, 2004 at 02:27:07PM -0600, Glenn English wrote:
I have Amanda configured to do all its backups (local and net) to disk.
I set the "reserved" parameter to 0 -- that will make it do complete
backups, right? (As opposed to degraded mode)

And if I run amflush I'll have complete backups on tape of everything in
the disklist, right?

No, the reserve parameter has no effect on whether level 0 backups
or incremental backups will be done.

Instead, it says, if a backup is level 0, can it be buffered on
the holding disk. If the reserve parameter is set to 100, then
100% of the holding disk space can be used only for incrementals.
All level 0's will go straight to tape. If the reserve is set to
25%, then that portion can only be used to buffer incrementals
while the other 75% of the holding disk can be used for either
level 0's or incrementals as needed.

jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)

Instead, it says, if a backup is level 0, can it be buffered on
the holding disk. If the reserve parameter is set to 100, then
100% of the holding disk space can be used only for incrementals.
All level 0's will go straight to tape. If the reserve is set to
25%, then that portion can only be used to buffer incrementals
while the other 75% of the holding disk can be used for either
level 0's or incrementals as needed.

$(prefix)/share/amanda/WHATS.NEW says that this applies only in
degraded mode. Before I put in 'reserve 50' I was seeing level 0s on
holding disk, but amanda would not _start_ any level 0s if the tape
was missing or had failed.

I do something similar to Glenn, but for a different reason. I do
'archive' runs on the weekend to holding disk, and flush them during
the day on Monday. This avoids backing up active filesystems (and the
slowness from that), and the tape drive is idle then. I set reserve
to 0 so I can use the whole holding disk - each week does 10G of level
0s, with skip-incr.

Quoting the docs:

* LEVEL-0 DUMPS ALLOWED WITH NO TAPE

If there is no tape present (or the tape drive fails during dumping),
Amanda switches to degraded mode. In degraded mode, level-0 dumps
are not allowed. This can be a pain for unattended sites over the
weekend (especially when there is a large holding disk that can hold
any necessary dumps). Amanda now supports a new configuration file
directive, "reserve". This tells Amanda to reserve that percentage
of total holding disk space for degraded mode dumps. Example: your
total holding disk space adds up to 8.4GB. If you specify a reserve
of 50, 4.2GB (50%) of the holding disk space will be allowed to be
used for regular dumps, but if that limit is hit, Amanda will switch
to degraded dumps. For backward compatibility, if no 'reserve' keyword
is present, 100 will be assumed (e.g. never do full dumps if degraded
mode is in effect). NOTE: this percentage applies from run to run, so,
as in the previous example, when Amanda runs the next day, if there is
3.8GB left on the holding disk, 1.9GB will be reserved for degraded
mode dumps (e.g. the percentage keeps sliding).

--
Greg Troxel <gdt < at > ir.bbn.com>

On Thu, Jun 17, 2004 at 08:02:59AM -0400, Greg Troxel wrote:
Instead, it says, if a backup is level 0, can it be buffered on
the holding disk. If the reserve parameter is set to 100, then
100% of the holding disk space can be used only for incrementals.
All level 0's will go straight to tape. If the reserve is set to
25%, then that portion can only be used to buffer incrementals
while the other 75% of the holding disk can be used for either
level 0's or incrementals as needed.

$(prefix)/share/amanda/WHATS.NEW says that this applies only in
degraded mode. Before I put in 'reserve 50' I was seeing level 0s on
holding disk, but amanda would not _start_ any level 0s if the tape
was missing or had failed.


Are you sure that the amount of holding disk space you have,
minus 50%, is sufficient to hold those level 0's?



The whats."NEW" is actually quite old. Note at the top of the
document it indicates these are "new" as of version 2.3 and about
40% of the way through the version 2.2 document is appended.
The section you quote later is from that appended section and
is, I believe, misleading.

The primary change I'd make in the language as it applies to this
this discussion is in this section:

If there is no tape present (or the tape drive fails during dumping),
Amanda switches to degraded mode. In degraded mode, level-0 dumps
are not allowed.

I would add the need to consider the amount of unreserved holding
disk space to the definition of degraded mode. Something like:

If a DLE is scheduled for a level-0 dump it is backed up in
degraded mode if there is no tape present (or the tape drive
fails during dumping) and there is insufficient unreserved
holding disk space to store the entire anticipated level-0 dump.

Note, I think (uncertaintly level high) that this decision is made
on a DLE by DLE basis, not on an amdump run by amdump run basis.

My personal experience is that level-0's quite happily get done
with my reserve of 30 percent and no tape present. In fact,
due to a sloppy, lazy admin, the tape magazine has not been
switched for 2 weeks and last night there were still many level-0
dumps done to the holding disk. The reason this works is that there
is still lots of holding disk space available.


--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
4455 Province Line Road (609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)

On Thu, 2004-06-17 at 08:03, Jon LaBadie wrote:

If there is no tape present (or the tape drive fails during dumping),
Amanda switches to degraded mode. In degraded mode, level-0 dumps
are not allowed.

Don't think so. According to the ChangeLog in the Amanda source:

1998-06-28 Alexandre Oliva <oliva < at > dcc.unicamp.br>

* server-src/reporter.c (output_tapeinfo):
degraded mode no longer
implies incremental dumps only.


--
Glenn English <ghe < at > slsware.com>

Are you sure that the amount of holding disk space you have,
minus 50%, is sufficient to hold those level 0's?

Yes. They dump and fit. I have 25G of holding space and only dump
10G or so.

The whats."NEW" is actually quite old. Note at the top of the
document it indicates these are "new" as of version 2.3 and about
40% of the way through the version 2.2 document is appended.
The section you quote later is from that appended section and
is, I believe, misleading.

Well, I didn't find anything else in the docs, so something may need
updating.

My personal experience is that level-0's quite happily get done
with my reserve of 30 percent and no tape present. In fact,
due to a sloppy, lazy admin, the tape magazine has not been
switched for 2 weeks and last night there were still many level-0
dumps done to the holding disk. The reason this works is that there
is still lots of holding disk space available.

Sure, that agrees with my experience. But if you have 'reserve 100',
or no statement (defaulting to old behavior), with no tape I would
expect no fulls to be done, but with a tape they will be.

--
Greg Troxel <gdt < at > ir.bbn.com>