Welcome! » Log In » Create A New Profile

VTape configuration

Posted by Chris Miller 
Chris Miller
VTape configuration
December 29, 2017 12:59PM
Hi Folks,

Just setting things up, and I think I ran out of docs...

I have no tape drives, so that means "vtape", which is fine, as long as I can size the vtape volume to be no larger than and no smaller than the current backup under consideration. Is this possible? Seems like a silly question, but physical tapes have a specific size and a small backup will result in surplus tape and a large backup will result in multiple volumes. I'd like to avoid both situations.

I found very little documentation. In fact there was so little, that I think I saw all of it this morning. I have found: http://wiki.zmanda.com/index.php/User_documentation Is there a hidden archive that I haven't found?
--
Chris.

V:916.974.0424
F:916.974.0428
This message was imported via the External PhorumMail Module
Winston Sorfleet
Re: VTape configuration
December 29, 2017 01:59PM
Hello Chris

Could you explain your concern about multiple volumes? You would normally set up a number of vtapes, and each amanda run would use one or more of them depending on the size of the backup for that particular run. For me I use 20GB volumes as it replaced an old DDS4 robot. I have about 40 in the pool (the NAS is 1TB) and each run uses anywhere between 2 and 6 volumes (the largest being when one of the client machines' partition get a full, and the smallest when a couple of client machines happen to get L2s lining up)

On 29 December 2017 15:28:33 GMT-05:00, Chris Miller <cjm@tryx.org> wrote:
>Hi Folks,
>
>Just setting things up, and I think I ran out of docs...
>
>I have no tape drives, so that means "vtape", which is fine, as long as
>I can size the vtape volume to be no larger than and no smaller than
>the current backup under consideration. Is this possible? Seems like a
>silly question, but physical tapes have a specific size and a small
>backup will result in surplus tape and a large backup will result in
>multiple volumes. I'd like to avoid both situations.
>
>I found very little documentation. In fact there was so little, that I
>think I saw all of it this morning. I have found:
>http://wiki.zmanda.com/index.php/User_documentation Is there a hidden
>archive that I haven't found?
>--
>Chris.
>
>V:916.974.0424
>F:916.974.0428

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
This message was imported via the External PhorumMail Module
Jon LaBadie
Re: VTape configuration
December 29, 2017 02:59PM
On Fri, Dec 29, 2017 at 12:28:33PM -0800, Chris Miller wrote:
> Hi Folks,
>
> Just setting things up, and I think I ran out of docs...
>
> I have no tape drives, so that means "vtape", which is fine, as long as I can size the vtape volume to be no larger than and no smaller than the current backup under consideration. Is this possible? Seems like a silly question, but physical tapes have a specific size and a small backup will result in surplus tape and a large backup will result in multiple volumes. I'd like to avoid both situations.
>
> I found very little documentation. In fact there was so little, that I think I saw all of it this morning. I have found: http://wiki.zmanda.com/index.php/User_documentation Is there a hidden archive that I haven't found?

Welcome to amanda Chris. Two comments first.

The size of a vtape is its "maximum" size. A 100GB vtape does not take
up 100GB of disk unless it contains a backup of that size.

Working with multiple vtapes and virtual changers is a breeze in amanda.
Don't try to avoid multiple tape backups.

Suppose you were used to working with 100GB physical tapes. You may
feel inclined to create 100GB vtapes in amanda. You may do that and
if your disk is 1TB, you would probably create 10 vtapes. Explore
2 possibilities.

Using 100GB vtapes, telling amanda to use only 1 vtape per run, you
may discover most (all?) your vtapes are not full. After you gain
some data from backups you may find your tapes average well under
100% full and you can allocate more vtapes than 10 x 100GB to your
1TB drive.

As a real example, I have 20TB of storage for vtapes and 240 x 100GB
vtapes. The 6 disks average 75-80% full even though I have 20% more
vtapes than I "should".

Another possibility is to allocate vtapes of 10 (or 20GB) and tell
amanda that it can use up to 10 (or 5) vtapes per run. Also tell
amanda that DLEs* may be split across multiple tapes using "chunks"
of about 10% or 20% the size of the vtape.

This would be an efficient use of your storage. A backup totalling
30GB would take 3 vtapes, 90GB would take 9. Each vtape except the
last would be filled to within 1 "chunk"-size.

Good Luck,
HTH,
Jon

* DLE == Disk List Entry, the unit of backup for amanda. A DLE
may be a file system (say / the root fs) or other more complex
entries (like root but not /var or /opt or /usr/local which are
backed up in other DLEs
--
Jon H. LaBadie jon@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (703) 935-6720 (C)
This message was imported via the External PhorumMail Module
Gene Heskett
Re: VTape configuration
December 29, 2017 04:59PM
On Friday 29 December 2017 17:12:07 Jon LaBadie wrote:

> On Fri, Dec 29, 2017 at 12:28:33PM -0800, Chris Miller wrote:
> > Hi Folks,
> >
> > Just setting things up, and I think I ran out of docs...
> >
> > I have no tape drives, so that means "vtape", which is fine, as long
> > as I can size the vtape volume to be no larger than and no smaller
> > than the current backup under consideration. Is this possible? Seems
> > like a silly question, but physical tapes have a specific size and a
> > small backup will result in surplus tape and a large backup will
> > result in multiple volumes. I'd like to avoid both situations.
> >
> > I found very little documentation. In fact there was so little, that
> > I think I saw all of it this morning. I have found:
> > http://wiki.zmanda.com/index.php/User_documentation Is there a
> > hidden archive that I haven't found?
>
> Welcome to amanda Chris. Two comments first.
>
> The size of a vtape is its "maximum" size. A 100GB vtape does not
> take up 100GB of disk unless it contains a backup of that size.
>
> Working with multiple vtapes and virtual changers is a breeze in
> amanda. Don't try to avoid multiple tape backups.
>
> Suppose you were used to working with 100GB physical tapes. You may
> feel inclined to create 100GB vtapes in amanda. You may do that and
> if your disk is 1TB, you would probably create 10 vtapes. Explore
> 2 possibilities.
>
> Using 100GB vtapes, telling amanda to use only 1 vtape per run, you
> may discover most (all?) your vtapes are not full. After you gain
> some data from backups you may find your tapes average well under
> 100% full and you can allocate more vtapes than 10 x 100GB to your
> 1TB drive.
>
> As a real example, I have 20TB of storage for vtapes and 240 x 100GB
> vtapes. The 6 disks average 75-80% full even though I have 20% more
> vtapes than I "should".
>
> Another possibility is to allocate vtapes of 10 (or 20GB) and tell
> amanda that it can use up to 10 (or 5) vtapes per run. Also tell
> amanda that DLEs* may be split across multiple tapes using "chunks"
> of about 10% or 20% the size of the vtape.
>
> This would be an efficient use of your storage. A backup totalling
> 30GB would take 3 vtapes, 90GB would take 9. Each vtape except the
> last would be filled to within 1 "chunk"-size.
>
> Good Luck,
> HTH,
> Jon
>
While we are on this subject, why, with a 10 day cycle, working over 30
vtapes of nominally 3.2 GB per vtape, does amanda's planner absolutely
refuse to use a level 2 or beyond? It often promotes a level 2 to a
level 0 when it has 6+ days left in its 10 day cycle.

The end result is a 1 TB drive with those 30 vtapes on it, stays at about
87-90% capacity. Currently doing this machine, 2 in the shop and 4 in
the garage every night.

I'd love to setup a 4 day cycle, but amanda refuses to co-operate.

Another is that the planner can setup a level 2, say so in the email I
get when its done, but 40 lines down in a 57 active line lde, when it
has actually done it, something is getting confused and a level 0 is
actually done to that DLE the planner said it was going to do a level 2
of.

Not a good way to run a train. But this year is into the 19th year I've
been using amanda.

Here is a furinstance
planner: Incremental of coyote:/GenesAmandaHelper-0.61 bumped to level 4.
coyote /GenesAmandaHelper-0.61 0 7084 3333 47.1 12:58 4387.5
0:35 97515.3

My logs are loaded with similar furinstances...

> * DLE == Disk List Entry, the unit of backup for amanda. A DLE
> may be a file system (say / the root fs) or other more complex
> entries (like root but not /var or /opt or /usr/local which are
> backed up in other DLEs


Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page http://geneslinuxbox.net:6309/gene
This message was imported via the External PhorumMail Module
Ned Danieley
amanda planner and level 0 promotion
December 30, 2017 05:59AM
On Fri, Dec 29, 2017 at 07:01:05PM -0500, Gene Heskett wrote:
> >
> While we are on this subject, why, with a 10 day cycle, working over 30
> vtapes of nominally 3.2 GB per vtape, does amanda's planner absolutely
> refuse to use a level 2 or beyond? It often promotes a level 2 to a
> level 0 when it has 6+ days left in its 10 day cycle.
....
>
>
> Cheers, Gene Heskett

and on the other side of the coin, is there any way to encourage the planner
to do more level 0 dumps in a given day?

I occasionally find, often after a kernel update/reboot, that amanda
thinks that a large number of DLE are 'new', and fairly quickly moves them up
to level 2. but then the planner leaves them at L2 for days or weeks, at the
same time using less than 1 TB of my 2.5 TB tape.

I know that I can force a level 0, but that has to be done on each DLE. it
seems like there should be a way to tell just amanda to use a larger
fraction of the tape, but I can't figure it out.

--
Ned Danieley (ned.danieley@duke.edu)
Department of Biomedical Engineering
Box 90281, Duke University
Durham, NC 27708 (919) 660-5111

http://dilbert.com/strips/comic/2012-02-11/
This message was imported via the External PhorumMail Module
Jon LaBadie
Re: amanda planner and level 0 promotion
December 30, 2017 12:59PM
On Sat, Dec 30, 2017 at 08:33:42AM -0500, Ned Danieley wrote:
>
> and on the other side of the coin, is there any way to encourage the planner
> to do more level 0 dumps in a given day?
>
> I occasionally find, often after a kernel update/reboot, that amanda
> thinks that a large number of DLE are 'new', and fairly quickly moves them up
> to level 2. but then the planner leaves them at L2 for days or weeks, at the
> same time using less than 1 TB of my 2.5 TB tape.
>
> I know that I can force a level 0, but that has to be done on each DLE. it
> seems like there should be a way to tell just amanda to use a larger
> fraction of the tape, but I can't figure it out.
>

As an ideal, amanda tries to get consistant size dumps
each run of amdump. As an approximation this size can be
estimated (using post-compression data) as the sum of
all level 0's divided by the number of runs per dumpcycle
plus the average daily level 1 sizes.

So you can get larger runs by three means, get more data
(bigger level 0's), become more active (bigger level 1's)
or shorten the dumpcycle (or runs/dumpcycle).

Sounds like you may want to shorten your dumpcycle.

Note, dumpcycle is a "dumptype" parameter. I.e. thought
there is a global default value for dumpcycle, new dump-
types can be defined with a different dumpcycle.

In my case I have my photos and my music backed up in
their own DLEs, 5 DLEs in the case of photos. These are
nearly static DLEs, so no need to back them up each week.
I defined a separate dumptype to have a 3 week dumpcycle
and these 6 DLEs use that dumptype.

You may want to define a dumptype for your more precious
DLEs with a shorter dumpcycle which will cause them to
be dumped at level 0 more frequently. Or just shorten
your default dumpcycle.

Jon
--
Jon H. LaBadie jon@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (703) 935-6720 (C)
This message was imported via the External PhorumMail Module
Gene Heskett
Re: amanda planner and level 0 promotion
December 30, 2017 01:59PM
On Saturday 30 December 2017 15:02:05 Jon LaBadie wrote:

> On Sat, Dec 30, 2017 at 08:33:42AM -0500, Ned Danieley wrote:
> > and on the other side of the coin, is there any way to encourage the
> > planner to do more level 0 dumps in a given day?
> >
> > I occasionally find, often after a kernel update/reboot, that amanda
> > thinks that a large number of DLE are 'new', and fairly quickly
> > moves them up to level 2. but then the planner leaves them at L2 for
> > days or weeks, at the same time using less than 1 TB of my 2.5 TB
> > tape.
> >
> > I know that I can force a level 0, but that has to be done on each
> > DLE. it seems like there should be a way to tell just amanda to use
> > a larger fraction of the tape, but I can't figure it out.
>
> As an ideal, amanda tries to get consistant size dumps
> each run of amdump. As an approximation this size can be
> estimated (using post-compression data) as the sum of
> all level 0's divided by the number of runs per dumpcycle
> plus the average daily level 1 sizes.
>
> So you can get larger runs by three means, get more data
> (bigger level 0's), become more active (bigger level 1's)
> or shorten the dumpcycle (or runs/dumpcycle).
>
> Sounds like you may want to shorten your dumpcycle.
>
> Note, dumpcycle is a "dumptype" parameter. I.e. thought
> there is a global default value for dumpcycle, new dump-
> types can be defined with a different dumpcycle.
>
> In my case I have my photos and my music backed up in
> their own DLEs, 5 DLEs in the case of photos. These are
> nearly static DLEs, so no need to back them up each week.
> I defined a separate dumptype to have a 3 week dumpcycle
> and these 6 DLEs use that dumptype.
>
> You may want to define a dumptype for your more precious
> DLEs with a shorter dumpcycle which will cause them to
> be dumped at level 0 more frequently. Or just shorten
> your default dumpcycle.
>
Since a 10 day cycle was no help, I had several weeks back, gone back to
a 7 day cycle. We'll see how that works with your PM'd settings.
I assume I should watch the df report, currently:
/dev/sdc2 960798056
668360948 243624668 74% /amandatapes

And that shows that commenting that stuff out helps, it was 88% a month
ago. We'll see which way it tends in the next week or 4. I can, at that
level of drive usage, put off swapping the 1TB for a 2TB I took out of a
milling machine when I put a 60GB SSD in it. Those things are FAST! No
mount, its just hanging on the cable. I need to round up some more of
those, and some 3.5" mount adapters. Even 60Gigs is overkill for a
milling machine unless using a code generator, as they generate
megabytes of code that I can write by hand that does the same thing in
60 lines of gcode. Probably to a higher accuracy. Both are better than
the machine, but...

> Jon

Have a better 2018, Jon, and thanks.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page http://geneslinuxbox.net:6309/gene
This message was imported via the External PhorumMail Module
Chris Miller
Re: VTape configuration
December 31, 2017 04:59PM
Hi Winston,

| Could you explain your concern about multiple volumes?
Yes. It adds a layer between the catalog of backed-up files and the location on disk. Without tape volumes, I could presumably find what I need with a date and tar, with tape volumes, I have the additional complexity of unpacking the tape volume discipline. If I could configure tape volumes so that each volume held exactly one backup, then I could live with that compromise.

Is there any documentation that I haven't found, because I haven't found much...

Thanks for the help,
--
Chris.

V:916.974.0424
F:916.974.0428
This message was imported via the External PhorumMail Module
Winston Sorfleet
Re: VTape configuration
December 31, 2017 06:59PM
Chris,

For the longest time I did traditional backups (fulls and incrementals,
via tar).  If that model still fits your needs, you can stay with that.

Once I began backing up a small cluster of machines, Amanda's paradigm
began to show its value.  However, it relies upon volume management, and
it requires an acceptance that "her" algorithms allow a more optimal
distribution of backups based on both availability of backup media
(virtual or otherwise) and the amount of individual and collective
changes on the client machines - as well as certain parameters I set
such as the maximum interval I am willing to accept between full
backups.  The benefit is that with the amount of space I've allocated to
vtapes, I get the maximum amount of change data on backup.  It isn't
overprovisioning; it's about optimization.  It's also proven itself in
restores, where instead of having to restore a full directory and then
every L1 and L2 delta, I can simply tell Amanda to restore
file-version-as-of-specific-date.

I highly suggest a read of this FAQ:
http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F;
particularly the section about Amanda's planning strategy.  If you
"insist" on constraining Amanda to one-volume-per-backup, you are
basically going against the strategy; without that capability, I don't
think that Amanda's overhead gives you anything you can't do with tar
and a cron job.


On 2017-12-31 07:13 PM, Chris Miller wrote:
> Hi Winston,
>
> Could you explain your concern about multiple volumes?
>
> Yes. It adds a layer between the catalog of backed-up files and the
> location on disk. Without tape volumes, I could presumably find what I
> need with a date and tar, with tape volumes, I have the additional
> complexity of unpacking the tape volume discipline. If I could
> configure tape volumes so that each volume held exactly one backup,
> then I could live with that compromise.
>
> Is there any documentation that I haven't found, because I haven't
> found much...
>
> Thanks for the help,
> --
> Chris.
>
> V:916.974.0424
> F:916.974.0428
This message was imported via the External PhorumMail Module
Ned Danieley
Re: amanda planner and level 0 promotion
January 01, 2018 11:59AM
On Sat, Dec 30, 2017 at 03:02:05PM -0500, Jon LaBadie wrote:
> On Sat, Dec 30, 2017 at 08:33:42AM -0500, Ned Danieley wrote:
> >
> > and on the other side of the coin, is there any way to encourage the planner
> > to do more level 0 dumps in a given day?
> >
> > I occasionally find, often after a kernel update/reboot, that amanda
> > thinks that a large number of DLE are 'new', and fairly quickly moves them up
> > to level 2. but then the planner leaves them at L2 for days or weeks, at the
> > same time using less than 1 TB of my 2.5 TB tape.
> >
> > I know that I can force a level 0, but that has to be done on each DLE. it
> > seems like there should be a way to tell just amanda to use a larger
> > fraction of the tape, but I can't figure it out.
> >
>
> As an ideal, amanda tries to get consistant size dumps
> each run of amdump. As an approximation this size can be
> estimated (using post-compression data) as the sum of
> all level 0's divided by the number of runs per dumpcycle
> plus the average daily level 1 sizes.
>
> So you can get larger runs by three means, get more data
> (bigger level 0's), become more active (bigger level 1's)
> or shorten the dumpcycle (or runs/dumpcycle).

okay, that's what I wanted to know. I'll try shortening the dumpcycle.
thanks

--
Ned Danieley (ned.danieley@duke.edu)
Department of Biomedical Engineering
Box 90281, Duke University
Durham, NC 27708 (919) 660-5111

http://dilbert.com/strips/comic/2012-02-11/
This message was imported via the External PhorumMail Module
Chris Miller
Re: VTape configuration
January 03, 2018 01:59PM
Hi Winston,

| For the longest time I did traditional backups (fulls and incrementals, via
| tar). If that model still fits your needs, you can stay with that.

| Once I began backing up a small cluster of machines, Amanda's paradigm began to
| show its value. However, it relies upon volume management, and it requires an
| acceptance that "her" algorithms allow a more optimal distribution of backups
| based on both availability of backup media (virtual or otherwise) and the
| amount of individual and collective changes on the client machines - as well as
| certain parameters I set such as the maximum interval I am willing to accept
| between full backups. The benefit is that with the amount of space I've
| allocated to vtapes, I get the maximum amount of change data on backup. It
| isn't overprovisioning; it's about optimization. It's also proven itself in
| restores, where instead of having to restore a full directory and then every L1
| and L2 delta, I can simply tell Amanda to restore
| file-version-as-of-specific-date.

| I highly suggest a read of this FAQ: [
| http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
| |
| http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
| ] ; particularly the section about Amanda's planning strategy. If you "insist"
| on constraining Amanda to one-volume-per-backup, you are basically going
| against the strategy; without that capability, I don't think that Amanda's
| overhead gives you anything you can't do with tar and a cron job.

I understand how Amanda wants to try to "smooth" the mix of backup levels and filesystem sizes so that backup costs about the same amount of time and storage each cycle, and that is a very worthwhile goal, so I don't want to impede that. I also understand that tape discipline is already built-in to Amanda at a fundamental level, so I don't want to mess with that, either. So, I seek advice.

Suppose "amanda.example.com" is backing-up "client.example.com" to "NAS0.example.com". My level 0 backups are typically 120GB and my level 1 backups are typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 150GB/week, meaning I have sufficient room for thirteen weeks of backup. This seems to me like it might be a pretty common scenario and that there might be example configs floating around that would size the vtapes for optimal use. Is there one?

I have some questions:

1. Can I make my vtapes all 150GB, and then instruct Amanda to put one cycle (one level 0, and six level 1) on one vtape, meaning re-use a vtape multiple time in a backup cycle? I like this approach quite a bit, if it is possible. It "packages" one level 0 with all the attendant level 1 differentials and eliminates my strongest reservations about vtapes -- namely that I don't know where anything is.
2. Failing that, should I make my vtapes 120GB, so I can fit a level 0 backup on one vtape, but then will Amanda truncate level 1 backups, so that the vtapes storage requirement is NOT 120G, but more like 5GB?
3. Alternatively, I could make my vtapes all 5GB and then Amanda will have to span fourteen vtapes for the level 0? This might be optimal use of storage, but it scares me with added complexity. I won't know where anything is, meaning I will have to have Amanda tools to unpack a backup, and in the case of a disaster, that may be really inconvenient.

I sure would like to have option 1, if I could...
--
Chris.

V:916.974.0424
F:916.974.0428
This message was imported via the External PhorumMail Module
Stefan G. Weichinger
Re: VTape configuration
January 03, 2018 01:59PM
Am 2018-01-03 um 22:08 schrieb Chris Miller:

> I have some questions:
>
> 1. Can I make my vtapes all 150GB, and then instruct Amanda to put one
> cycle (one level 0, and six level 1) on one vtape, meaning re-use a
> vtape multiple time in a backup cycle? I like this approach quite a
> bit, if it is possible. It "packages" one level 0 with all the
> attendant level 1 differentials and eliminates my strongest
> reservations about vtapes -- namely that I don't know where anything is.
> 2. Failing that, should I make my vtapes 120GB, so I can fit a level 0
> backup on one vtape, but then will Amanda truncate level 1 backups,
> so that the vtapes storage requirement is NOT 120G, but more like 5GB?
> 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
> have to span fourteen vtapes for the level 0? This might be optimal
> use of storage, but it scares me with added complexity. I won't know
> where anything is, meaning I will have to have Amanda tools to
> unpack a backup, and in the case of a disaster, that may be really
> inconvenient.
>
> I sure would like to have option 1, if I could...

I would say, go with option 2 if you want to avoid complexity.
The vtapes won't all use up the full defined size if the actual daily
dump needs less space. So why not specify that a bit larger than the
expected lev0/FULL-run and see how that works out?

AFAIK option 1 is not possible with current Amanda-strategies (at least
not easily or without other complexities involved). But maybe I am wrong
here ...
This message was imported via the External PhorumMail Module
Jean-Louis Martineau
Re: VTape configuration
January 03, 2018 02:59PM
A vtape do not reserve the space, it only use the space of the dump you
put on it.
The vtape size should be the maximum size of any days or larger, they
could be 120GB or 2TG, the result will be the same.
Some vtape will use 5GB, some will use 120GB.

Jean-Louis

On 03/01/18 04:08 PM, Chris Miller wrote:
> Hi Winston,
>
> For the longest time I did traditional backups (fulls and
> incrementals, via tar). If that model still fits your needs, you
> can stay with that.
>
> Once I began backing up a small cluster of machines, Amanda's
> paradigm began to show its value. However, it relies upon volume
> management, and it requires an acceptance that "her" algorithms
> allow a more optimal distribution of backups based on both
> availability of backup media (virtual or otherwise) and the amount
> of individual and collective changes on the client machines - as
> well as certain parameters I set such as the maximum interval I am
> willing to accept between full backups. The benefit is that with
> the amount of space I've allocated to vtapes, I get the maximum
> amount of change data on backup. It isn't overprovisioning; it's
> about optimization. It's also proven itself in restores, where
> instead of having to restore a full directory and then every L1
> and L2 delta, I can simply tell Amanda to restore
> file-version-as-of-specific-date.
>
> I highly suggest a read of this FAQ:
> http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
> http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F;
> particularly the section about Amanda's planning strategy. If you
> "insist" on constraining Amanda to one-volume-per-backup, you are
> basically going against the strategy; without that capability, I
> don't think that Amanda's overhead gives you anything you can't do
> with tar and a cron job.
>
>
> I understand how Amanda wants to try to "smooth" the mix of backup
> levels and filesystem sizes so that backup costs about the same amount
> of time and storage each cycle, and that is a very worthwhile goal, so
> I don't want to impede that. I also understand that tape discipline is
> already built-in to Amanda at a fundamental level, so I don't want to
> mess with that, either. So, I seek advice.
>
> Suppose "amanda.example.com
> http://amanda.example.com"
> is backing-up "client.example.com
> http://client.example.com"
> to "NAS0.example.com
> http://NAS0.example.com".
> My level 0 backups are typically 120GB and my level 1 backups are
> typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 150GB/week,
> meaning I have sufficient room for thirteen weeks of backup. This
> seems to me like it might be a pretty common scenario and that there
> might be example configs floating around that would size the vtapes
> for optimal use. Is there one?
>
> I have some questions:
>
> 1. Can I make my vtapes all 150GB, and then instruct Amanda to put
> one cycle (one level 0, and six level 1) on one vtape, meaning
> re-use a vtape multiple time in a backup cycle? I like this
> approach quite a bit, if it is possible. It "packages" one level 0
> with all the attendant level 1 differentials and eliminates my
> strongest reservations about vtapes -- namely that I don't know
> where anything is.
> 2. Failing that, should I make my vtapes 120GB, so I can fit a level
> 0 backup on one vtape, but then will Amanda truncate level 1
> backups, so that the vtapes storage requirement is NOT 120G, but
> more like 5GB?
> 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
> have to span fourteen vtapes for the level 0? This might be
> optimal use of storage, but it scares me with added complexity. I
> won't know where anything is, meaning I will have to have Amanda
> tools to unpack a backup, and in the case of a disaster, that may
> be really inconvenient.
>
> I sure would like to have option 1, if I could...
> --
> Chris.
>
> V:916.974.0424
> F:916.974.0428
This message is the property of CARBONITE, INC. and may contain confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to anyone. Instead, destroy it and notify me by reply e-mail
This message was imported via the External PhorumMail Module
Chris Miller
Re: VTape configuration
January 03, 2018 02:59PM
Hi Jean-Louis,

| A vtape do not reserve the space, it only use the space of the dump you put on
| it.
| The vtape size should be the maximum size of any days or larger, they could be
| 120GB or 2TG, the result will be the same.
| Some vtape will use 5GB, some will use 120GB.

Thanks very much. That is quite helpful.

Is it possible to re-use a vtape for several successive backups? If the vtape were twice the size of a level 0 backup, then the level 1 backups would be appended, but the next level 0 would be too large and would trigger a new tape.

Additionally, How do I configure larger cycles, meaning, I will use 150GB each week, and 2TB can hold 13 weeks. How do I tell Amanda that after 13 weeks, she should start over by deleting the oldest vtape?

Thanks for the help,
--
Chris.

V:916.974.0424
F:916.974.0428
This message was imported via the External PhorumMail Module
Winston Sorfleet
Re: VTape configuration
January 03, 2018 03:59PM
On 2018-01-03 05:46 PM, Chris Miller wrote:
> Hi Jean-Louis,
>
> A vtape do not reserve the space, it only use the space of the
> dump you put on it.
> The vtape size should be the maximum size of any days or larger,
> they could be 120GB or 2TG, the result will be the same.
> Some vtape will use 5GB, some will use 120GB.
>
> Thanks very much. That is quite helpful.
>
> Is it possible to re-use a vtape for several successive backups? If
> the vtape were twice the size of a level 0 backup, then the level 1
> backups would be appended, but the next level 0 would be too large and
> would trigger a new tape.
>
> Additionally, How do I configure larger cycles, meaning, I will use
> 150GB each week, and 2TB can hold 13 weeks. How do I tell Amanda that
> after 13 weeks, she should start over by deleting the oldest vtape?

That's exactly what it will do, unless you explicitly tell Amanda that a
tape is out of commission.  So, just like your traditional robot, it
will cycle.

HOWEVER, note that Amanda will not write to a tape that has been
recently written, whether it's 5 GB or 150 GB.  It does /not/ append to
tapes (barring some painful and unnecessary manual configuration, that
is).  So, you're better off using small vtapes.  Because if it only
writes 5 GB to 150 GB in a 2 TB NAS, that remaining 145 GB is
unavailable to Amanda until that vtape comes up for overwriting.


>
> Thanks for the help,
> --
> Chris.
>
> V:916.974.0424
> F:916.974.0428
This message was imported via the External PhorumMail Module
Jon LaBadie
Re: VTape configuration
January 03, 2018 03:59PM
On Wed, Jan 03, 2018 at 01:08:37PM -0800, Chris Miller wrote:
> Hi Winston,
>
> | I highly suggest a read of this FAQ: [
> | http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
> | ] ; particularly the section about Amanda's planning strategy. If you "insist"
> | on constraining Amanda to one-volume-per-backup, you are basically going
> | against the strategy; without that capability, I don't think that Amanda's
> | overhead gives you anything you can't do with tar and a cron job.
>
> I understand how Amanda wants to try to "smooth" the mix of backup
> levels and filesystem sizes so that backup costs about the same amount
> of time and storage each cycle, and that is a very worthwhile goal,
> so I don't want to impede that. I also understand that tape discipline
> is already built-in to Amanda at a fundamental level, so I don't want
> to mess with that, either. So, I seek advice.
>
> Suppose "amanda.example.com" is backing-up "client.example.com" to
> "NAS0.example.com". My level 0 backups are typically 120GB and my level 1
> backups are typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 150GB/week,
> meaning I have sufficient room for thirteen weeks of backup. This seems to me
> like it might be a pretty common scenario and that there might be example
> configs floating around that would size the vtapes for optimal use. Is there one?

I'm not sure you do understand amanda's approach Chris. Amanda does not
try to smooth things across a dump cycle, but across each run within a
dump cycle. If your dumpcycle is 7 days with daily amdump runs, amanda's
goal would be 5GB incrementals plus 120GB/7 (about 18GB) for a daily
dump of about 23GB.

That ideal might be achievable if you had about 14 DLE averaging about
5GB each. Then amanda could decide which 1,2,3, or 4 DLEs to do level 0
dumps each run. But in real life you have some small and some large
DLEs and amanda can't meet that ideal. Assuming you have at least a
few DLEs in those 120GB, some days might have a 60GB level 0 DLE to
dump, other days maybe a 5 and a 10 GB, and still others maybe none
and you will only have incrementals.

In such a scenario I would try to size my vtapes so the largest (60GB
in my example) would fit comfortably in about 3 tapes including a set
of incrementals. So something like 25GB/vtape and tell amanda it can
use up to 3 tapes per amdump run. The day that large DLE gets a full
dump, 3 tapes will be used. Most other days probably only 1 tape will
be needed.

> I have some questions:

Again, based on the questions below, you don't yet get the amanda scheme.
There will not be A (as in "a single") level 0, each DLE will get its
own schedule of level 0's.

Amanda is nothing if not flexible. You can force it to do things in a
"non-amanda" way. My comments after each question are about how you
might achieve the goal, but then why use amanda? .
>
> 1. Can I make my vtapes all 150GB, and then instruct Amanda to put
> one cycle (one level 0, and six level 1) on one vtape, meaning re-use a
> vtape multiple time in a backup cycle? I like this approach quite a bit,
> if it is possible. It "packages" one level 0 with all the attendant
> level 1 differentials and eliminates my strongest reservations about
> vtapes -- namely that I don't know where anything is.

Amanda will not write a new amdump to the end of an existing tape containing
a previous dump.

I've not seen mention of a "holding disk" in this thread. If you are using
one (strongly recommended) and it is sufficiently large, you could run each
amdump with the option to not write to tape. The dumps will then collect
on the holding disk. Then one day a week have your cronjob also do an
amflush which will move all the dumps, full and incr., to one or more vtapes.

You could even do your traditional all level 0's the same day if you
want. Specify a dumptype of incremental only and on the day you choose
specify a forced full dump.

> 2. Failing that, should I make my vtapes 120GB, so I can fit a
> level 0 backup on one vtape, but then will Amanda truncate level 1 backups,
> so that the vtapes storage requirement is NOT 120G, but more like 5GB?

You could make a 100 120GB or 150GB vtapes. They are just directories
and take no space. Their size is the maximum amount of data amanda will
write to a single vtape. So if only 10GB get written to each tape it
those 100 tapes will only take 1/2 of your 2TB space on NAS0.

Amanda will not truncate any dumps. Again assuming you are using a
holding disk, any dumps that don't fit the available vtape(s) will
remain on the holding disk until flushed to a vtape. This can happen
automatically on the next amdump run.

> 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
> have to span fourteen vtapes for the level 0? This might be optimal use of
> storage, but it scares me with added complexity. I won't know where anything
> is, meaning I will have to have Amanda tools to unpack a backup, and in the
> case of a disaster, that may be really inconvenient.

Hopefully you realize by now there is no "level 0".

And no, you do not need amanda tools to unpack a backup. It is one of
the things that attracted me to amanda. On one job I had been using a
commercial backup system. During a hardware upgrade we also upgraded
the backup software. After the old hardware and software was retired
we found that the new version could not recover old backups. We had
2 years of unrecoverable backups. With amanda backups I just need ls,
dd, tar, and gzip to recover things.

This may not mollify your fear of "knowing where things are" but you
get an email following each run telling you what went on. You can
also have amanda send to a printer a listing of each file it created
on which vtapes that run, i.e. a table of contents.

But ls works also, here is a listing of my tape 13.

# ls slot13
00000.DS1-013 00013.cyber.jgcomp.com.Vault-Monthly.1
00001.cyber.jgcomp.com.Photos-3.1 00014.cyber.jgcomp.com.Root.0
00002.cyber.jgcomp.com.Photos-1.1 00015.cyber.jgcomp.com.Root.0
00003.catwoman.jgcomp.com.E__Public.0 00016.mums.jgcomp.com.Root.0
00004.cyber.jgcomp.com.Photos-Misc.1 00017.mums.jgcomp.com.Root.0
00005.cyber.jgcomp.com.Photos-4.1 00018.cyber.jgcomp.com.Boot.0
00006.cyber.jgcomp.com.Photos-2.1 00019.mums.jgcomp.com.Boot.0
00007.cyberwin8.jgcomp.com.C__Users.1 00020.cyber.jgcomp.com.Var.1
00008.cyber.jgcomp.com.Home.1 00021.catwoman.jgcomp.com.C__Users.0
00009.cyber.jgcomp.com.JonMusic.1 00022.cyber.jgcomp.com.Vault-Daily.1
00010.cyber.jgcomp.com.Vault-Weekly.1 00023.cyber.jgcomp.com.Vault-Daily.1
00011.cyber.jgcomp.com.Vault-Current.1 00024.cyber.jgcomp.com.Local.0
00012.cyber.jgcomp.com.Vault-Monthly.1 00025.cyber.jgcomp.com.Opt.0

The format is:

00025. tape file number
cyber.jgcomp.com. host that was backed up
Opt. DLE name
0 Backup level

I could peek in file 00000 to be certain when the dump was done,
but probably the time stamps on the files or the directory are enough.

# ls -Lld slot13
drwxr-xr-x. 2 amandabackup disk 4096 Dec 23 03:37 slot13

Tape 13 contains the backups from 2 days before Christmas.

>
> I sure would like to have option 1, if I could...

As noted, you can. But then why use amanda?

Jon
--
Jon H. LaBadie jon@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (703) 935-6720 (C)
This message was imported via the External PhorumMail Module
Jon LaBadie
Re: VTape configuration
January 03, 2018 03:59PM
On Wed, Jan 03, 2018 at 06:22:41PM -0500, Winston Sorfleet wrote:
> On 2018-01-03 05:46 PM, Chris Miller wrote:
> > Hi Jean-Louis,
> >
> > A vtape do not reserve the space, it only use the space of the
> > dump you put on it.
> > The vtape size should be the maximum size of any days or larger,
> > they could be 120GB or 2TG, the result will be the same.
> > Some vtape will use 5GB, some will use 120GB.
> >
> > Thanks very much. That is quite helpful.
> >
> > Is it possible to re-use a vtape for several successive backups? If
> > the vtape were twice the size of a level 0 backup, then the level 1
> > backups would be appended, but the next level 0 would be too large and
> > would trigger a new tape.
> >
> > Additionally, How do I configure larger cycles, meaning, I will use
> > 150GB each week, and 2TB can hold 13 weeks. How do I tell Amanda that
> > after 13 weeks, she should start over by deleting the oldest vtape?
>
> That's exactly what it will do, unless you explicitly tell Amanda that a
> tape is out of commission.  So, just like your traditional robot, it
> will cycle.
>

One parameter you specify is how many tapes (minimum) you have in
rotation. Amanda will not reuse any tape until it has used that
many tapes.

I presently have 240 vtapes of 100GB each spread over 6 hard disks.
I have specified my "tapecycle" as 140, not 240. All 240 tapes are
used in sequence and then things rotate back to the beginning. But
the lower number is my protection against one of the disks failing
and amanda not having anywhere to put new backups.

> HOWEVER, note that Amanda will not write to a tape that has been
> recently written, whether it's 5 GB or 150 GB.  It does /not/ append to
> tapes (barring some painful and unnecessary manual configuration, that
> is).  So, you're better off using small vtapes.  Because if it only
> writes 5 GB to 150 GB in a 2 TB NAS, that remaining 145 GB is
> unavailable to Amanda until that vtape comes up for overwriting.
>
Perhaps I misunderstand Winston's comment here. But if you write 5GB
to a "150GB" vtape, the "remaining 145GB" are available for other vtapes.
My 240 x 100GB vtapes (nominally 24TB) are on 20TB of disk. None of the
six disks are over 80% full.

jon
--
Jon H. LaBadie jon@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (703) 935-6720 (C)
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login