Backup Central

Hi!

I want to setup amanda to backup some important data, but I am afraid
that I do not understand correctly how amanda works inside. I am not
sure if I am doing right or in a best practice way.

As the amount of data is very large (630GB), thus the backup time would
be very long (especially L0's) I divided the data into data which
changes fast and should be backed up daily (DailySet1) and data which is
very static and should be backed up weekly. The weekly data I also
divided into 4 parts (WeeklySet1-4) so that the runtime of the backup is
shorter.

The parameters for DailySet1 are:

dumpcycle 4 weeks
runspercycle 28
tapecycle 30 tapes
runtapes 4

tapelength 10 gbytes
tape_splitsize 1000 mbytes

The parameters for the WeeklySet1-4 are:

dumpcycle 4 weeks
runspercycle 4
tapecycle 6 tapes
runtapes 4

tapelength 50 gbytes
tape_splitsize 10000 mbytes

I am using Amanda 2.6.1p1-2 on a Ubuntu Lucid distro.

The largest DLE in the daily set is about 30G, that's the reason why I
used 10G tapes and let amanda run 4 tapes. I split the tapes into 1G
chunks to fill the tapes better.

The largest DLE in the weekly sets is about 180GB, that's the reason why
I used 50G tapes and let amanda run 4 tapes. I split the tapes into 10G
chunks to fill the tapes better.

I chose to use 30 tapes * 10GB + (6 tapes * 50GB) * 4 to fit the disk
space of 1,8T.

The keep time of the data does not need to be very long (I do not need
many old versions of specific data). It should be enough to have 2 week
old versions of the backed up data (two runs in WeeklySets and 14 runs
in DailySet).
In my understanding I would need at least L0 dumps every two weeks. To
keep the amount of data small and the time to backup short the amount of
incrementals should be as large as possible.

How do I accomplish all my needs with the resources I have?
Is the way I am thinking completely nonsense?
Are there any best practice rules tipps or recommendations for me?

Beyond I attach my configurations. I removed some default lines, not
needed dumptypes and disktypes and the comments for better readability.
Also I attach the disklists for all my sets.

I hope that anyone can help me

Any comments are appreciated. Thanks in advance!

Cheers,
Thomas

-----------------------------------------------------------------------
DailySet1 - disklist (runs daily) - about 50G:
----------------------------------
store01 Dokumente /storage/dokumente client-tar-compress-span # 10G(growing)
store01 WindowsHomes /storage/winhome client-tar-compress-span # 30G
store01 Scratch /storage/scratch client-tar-compress-span # 1G
store01 RootHome /root client-tar-compress-span # <1G
store01 KonfigFiles /etc client-tar-compress-span # <1G

hochschwab Mailserver /srv/mail client-tar-encrypt-compress-span # 2G
hochschwab RootHome /root client-tar-encrypt-compress-span # <1G
hochschwab KonfigFiles /etc client-tar-encrypt-compress-span # <1G
hochschwab Backups /var/backup client-tar-encrypt-compress-span # <1G

riegerin RootHome /root client-tar-compress-span # <1G
riegerin KonfigFiles /etc client-tar-compress-span # <1G

calendar Roothome /root client-tar-compress-span # <1G
calendar KonfigFiles /etc client-tar-compress-span # <1G
calendar Kalender /var/spool/caldavd client-tar-compress-span # <1G

store02 RootHome /root client-tar-compress-span # <1G
store02 KonfigFiles /etc client-tar-compress-span # <1G
---------------------------------------------------------------------------------------
WeeklySet1 - disklist (runs Mondays) - about 180G and growing:
-------------------------------------
store01 /storage/bilder client-tar-compress-span # 180G(growing)
---------------------------------------------------------------------------------------
WeeklySet2 - disklist (runs Tuesdays) - about 105G and growing:
-------------------------------------
store02 //macmini/Medien/Musik client-tar-compress-span # 100G
hochschwab /srv/www/berge client-tar-encrypt-compress-span # 5G(growing)
---------------------------------------------------------------------------------------
WeeklySet3 - disklist (runs Wednesdays) - about 190G:
----------------------------------------
store01 /storage/data/Amiga client-tar-compress-span # <1G
store01 /storage/data/MacOSX client-tar-compress-span # 60G
store01 /storage/data/Windows client-tar-compress-span # 130G
---------------------------------------------------------------------------------------
WeeklySet4 - disklist (runs Thursdays) - about 111G:
---------------------------------------
store01 /storage/data/Hardware client-tar-compress-span # 50G
store01 /storage/data/Karten client-tar-compress-span # 18G
store01 /storage/data/Scripte client-tar-compress-span # minimal
store01 /storage/data/Tutorials client-tar-compress-span # 43G
---------------------------------------------------------------------------------------
DailySet1 - amanda.conf:
------------------------
inparallel 4
dumporder "sssS"
taperalgo largestfit
displayunit "m"
usetimestamps yes
dumpcycle 4 weeks
runspercycle 28
tapecycle 30 tapes
bumpsize 20 Mb
bumppercent 10
bumpdays 1
bumpmult 4
etimeout 300
dtimeout 1800
ctimeout 30
runtapes 4
tpchanger "chg-disk"
tapedev "file:/storage/amanda/tapes/DailySet1"
changerfile "/etc/amanda/DailySet1/changer"
changerdev " < at > DEFAULT_CHANGER_DEVICE < at > "
tapetype DISK
labelstr "^DailySet1-[0-9][0-9]*$"
amrecover_do_fsf yes
amrecover_check_label yes
amrecover_changer " < at > DEFAULT_TAPE_DEVICE < at > "
autoflush no

define tapetype DISK {
comment "Backup to HDD"
length 10 gbytes
}
define dumptype global {
comment "Global definitions"
index yes
record no
split_diskbuffer "/tmp"
fallback_splitsize 64m
}

define dumptype client-tar-encrypt-compress-span {
global
comment "TAR with client compression and encryption and tape spanning"
compress client fast
strategy standard
priority high
encrypt client
client_encrypt "/usr/sbin/amcrypt"
client_decrypt_option "-d"
program "GNUTAR"
tape_splitsize 1000 mbytes # chunksize of dumps on tape
}
define dumptype client-tar-compress-span {
global
comment "TAR with client compression and tape spanning"
compress client fast
strategy standard
priority high
program "GNUTAR"
tape_splitsize 1000 mbytes # chunksize of dumps on tape
}

define interface local {
comment "a local disk"
}

define interface eth0 {
comment "1000 Mbps ethernet"
}
---------------------------------------------------------------------------
WeeklySet1 - amanda.conf:
-------------------------
inparallel 4
dumporder "sssS"
taperalgo largestfit
displayunit "m"
usetimestamps yes
dumpcycle 4 weeks
runspercycle 4
tapecycle 6 tapes
bumpsize 20 Mb
bumppercent 10
bumpdays 1
bumpmult 4
etimeout 300
dtimeout 1800
ctimeout 30
runtapes 4
tpchanger "chg-disk"
tapedev "file:/storage/amanda/tapes/WeeklySet1"
changerfile "/etc/amanda/WeeklySet1/changer"
changerdev " < at > DEFAULT_CHANGER_DEVICE < at > "
tapetype DISK
labelstr "^WeeklySet1-[0-9][0-9]*$"
amrecover_do_fsf yes
amrecover_check_label yes
amrecover_changer " < at > DEFAULT_TAPE_DEVICE < at > "
autoflush no

define tapetype DISK {
comment "Backup to HDD"
length 50 gbytes
}

define dumptype client-tar-encrypt-compress-span {
global
comment "TAR with client compression and encryption and tape spanning"
compress client fast
strategy standard
priority high
encrypt client
client_encrypt "/usr/sbin/amcrypt"
client_decrypt_option "-d"
program "GNUTAR"
tape_splitsize 10000 mbytes # chunksize of dumps on tape
}
define dumptype client-tar-compress-span {
global
comment "TAR with client compression and tape spanning"
compress client fast
strategy standard
priority high
program "GNUTAR"
tape_splitsize 10000 mbytes # chunksize of dumps on tape
}

define interface local {
comment "a local disk"
}

define interface eth0 {
comment "1000 Mbps ethernet"
}
----------------------------------------------------------------------------------------------------
WeeklySet2-4 - amanda.conf: same as WeeklySet1 just keyword WeeklySet1
replaced by the correct word.
----------------------------------------------------------------------------------------------------
Space on /storage where the tapes reside: 1,8T
----------------------------------------------------------------------------------------------------

On Wed, Oct 20, 2010 at 12:10:16PM +0200, Thomas Marko wrote:
Hi!

I want to setup amanda to backup some important data, but I am afraid
that I do not understand correctly how amanda works inside. I am not
sure if I am doing right or in a best practice way.

As the amount of data is very large (630GB), thus the backup time would
be very long (especially L0's) I divided the data into data which
changes fast and should be backed up daily (DailySet1) and data which is
very static and should be backed up weekly. The weekly data I also
divided into 4 parts (WeeklySet1-4) so that the runtime of the backup is
shorter.

I'm not sure if it is best to make up 5 distinct Amanda configs, or just
one config with customized DLEs and dumptypes. I'd probably try the
latter first. I think it would make the best use of your storage media
rather than targeting specific amounts for each config.

Maybe others will add their views.

There are a few things I will comment on below:


The parameters for DailySet1 are:

dumpcycle 4 weeks
runspercycle 28
tapecycle 30 tapes
runtapes 4

tapelength 10 gbytes
tape_splitsize 1000 mbytes

You are only asking for a single lvl 0 dump every 4 weeks. Dangerous
in my view. The days the lvl 0 is done for Documente and WindowsHomes
would take 2 and 4 tapes respectively. Some other days may also take
multiple tapes. Your 30 tapes will be used before your dumpcycle is
finished. And even if you add just a couple of tapes, it is likely
that the next lvl 0 will overwrite the older one. If anything happens,
(eg. network connection lost, system crash, disk errors, etc.)
you will have no lvl 0 remaining.

You don't seem to be using a holding disk so you will be writing
directly to the vtapes. That increases the likelyhood of trashing
something on problems. It also eliminates dumping multiple DLEs
in parallel. Rather one after the next will have to run.


The parameters for the WeeklySet1-4 are:

dumpcycle 4 weeks
runspercycle 4
tapecycle 6 tapes
runtapes 4

tapelength 50 gbytes
tape_splitsize 10000 mbytes


Mostly the same comments about the number of tapes.

Even if you want Daily and Weekly to be different configs, rather
than DLEs of one config, consider putting the DLEs from the four
weekly configs into a single config. First benefit you get is
24 tapes of the current size. The dumptype could be set to
"strategy nofull" (or is it "strategy incronly" ?) and use
amadmin "force" commands (from cron) to specify when to do fulls.

I am using Amanda 2.6.1p1-2 on a Ubuntu Lucid distro.

Seriously consider moving to the 3.x versions.


The largest DLE in the daily set is about 30G, that's the reason why I
used 10G tapes and let amanda run 4 tapes. I split the tapes into 1G
chunks to fill the tapes better.

The largest DLE in the weekly sets is about 180GB, that's the reason why
I used 50G tapes and let amanda run 4 tapes. I split the tapes into 10G
chunks to fill the tapes better.

I chose to use 30 tapes * 10GB + (6 tapes * 50GB) * 4 to fit the disk
space of 1,8T.

The keep time of the data does not need to be very long (I do not need
many old versions of specific data). It should be enough to have 2 week
old versions of the backed up data (two runs in WeeklySets and 14 runs
in DailySet).

In my understanding I would need at least L0 dumps every two weeks. To
keep the amount of data small and the time to backup short the amount of
incrementals should be as large as possible.

If you want lvl 0's every two weeks, then should to set the dumpcycle
to two weeks and the runspercycle appropriately.


How do I accomplish all my needs with the resources I have?
Is the way I am thinking completely nonsense?
Are there any best practice rules tipps or recommendations for me?

Beyond I attach my configurations. I removed some default lines, not
needed dumptypes and disktypes and the comments for better readability.
Also I attach the disklists for all my sets.

I hope that anyone can help me

Any comments are appreciated. Thanks in advance!

If you run your backups on the "ragged edge" of sufficient storage,
I would say your (i.e. your mgmt) do not value highly the data
that is being backed up. An extra pair of 1-2TB drives would cost
about $125-$250. Isn't extra safety of your data worth it?

jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

Thomas,

Not sure where to begin, so I will try to begin at the beginning.

Amanda is designed to run daily, for purposes of my site that
usually means 5 days per calendar week, since we don't have
staff to change tapes on the weekend and only some of our
tape drives are in jukeboxes/libraries.

Amanda is not normally run more than once/day, though it can,
separate discussion for another time.

Amanda wants to backup each of the DLEs, DiskList Entries, in
the disklist file at each run. As amanda gets the lay of the
land it begins to schedule the dump levels so as to try and
backup similar volumes of data in each run.

Amanda will dump level 0 _at least_ once every dumpcycle days.
For most configs at my site, dumpcycles is "weekly" meaning
I get at least one level zero of each file system each week.

We typically have a tapecycle, of 20 tapes. This gives me a
4 week tape rotation with at least 4 level 0 dumps and the
rest are at higher dump levels.

The tape savings, the, don't waste a lot of tape on static
files is handled for me by the dump levels as scheduled
by the amanda server. [Discusson how how it does that
withheld for another time, see 'estimate phase']

There is no longer any
- "select a system for fulls M and Thursday and
other systems T and Friday",
all of that static control is usually released to Amanda
and freeded by you. This sometimes takes takes a bit of
getting used to.

All of this takes place for a given amanda server (has amanda
work/spooling area and an output unit, tape drive or library
of some type) and a set of clients.

We have multiple amanda servers with non-overlapping client
pools but not because of scheduling. We have multiple amanda
servers because of 1) server/tape capacity 2) geographic location
3) political and/or funding domains 4) security requirements.

I have an amanda server that has little on it other than an OS
and samba shares, that system has 200+ gig worth of samba shares
but the nightly output to tape is less than 30 Gig per day.

Its a lot to take in, please feel free to ask additional questions.

Brian


On Thu, Oct 21, 2010 at 03:50:34PM -0400, Jon LaBadie wrote:
On Wed, Oct 20, 2010 at 12:10:16PM +0200, Thomas Marko wrote:
Hi!

I want to setup amanda to backup some important data, but I am afraid
that I do not understand correctly how amanda works inside. I am not
sure if I am doing right or in a best practice way.

As the amount of data is very large (630GB), thus the backup time would
be very long (especially L0's) I divided the data into data which
changes fast and should be backed up daily (DailySet1) and data which is
very static and should be backed up weekly. The weekly data I also
divided into 4 parts (WeeklySet1-4) so that the runtime of the backup is
shorter.

I'm not sure if it is best to make up 5 distinct Amanda configs, or just
one config with customized DLEs and dumptypes. I'd probably try the
latter first. I think it would make the best use of your storage media
rather than targeting specific amounts for each config.

Maybe others will add their views.

There are a few things I will comment on below:


The parameters for DailySet1 are:

dumpcycle 4 weeks
runspercycle 28
tapecycle 30 tapes
runtapes 4

tapelength 10 gbytes
tape_splitsize 1000 mbytes

You are only asking for a single lvl 0 dump every 4 weeks. Dangerous
in my view. The days the lvl 0 is done for Documente and WindowsHomes
would take 2 and 4 tapes respectively. Some other days may also take
multiple tapes. Your 30 tapes will be used before your dumpcycle is
finished. And even if you add just a couple of tapes, it is likely
that the next lvl 0 will overwrite the older one. If anything happens,
(eg. network connection lost, system crash, disk errors, etc.)
you will have no lvl 0 remaining.

You don't seem to be using a holding disk so you will be writing
directly to the vtapes. That increases the likelyhood of trashing
something on problems. It also eliminates dumping multiple DLEs
in parallel. Rather one after the next will have to run.


The parameters for the WeeklySet1-4 are:

dumpcycle 4 weeks
runspercycle 4
tapecycle 6 tapes
runtapes 4

tapelength 50 gbytes
tape_splitsize 10000 mbytes


Mostly the same comments about the number of tapes.

Even if you want Daily and Weekly to be different configs, rather
than DLEs of one config, consider putting the DLEs from the four
weekly configs into a single config. First benefit you get is
24 tapes of the current size. The dumptype could be set to
"strategy nofull" (or is it "strategy incronly" ?) and use
amadmin "force" commands (from cron) to specify when to do fulls.

I am using Amanda 2.6.1p1-2 on a Ubuntu Lucid distro.

Seriously consider moving to the 3.x versions.


The largest DLE in the daily set is about 30G, that's the reason why I
used 10G tapes and let amanda run 4 tapes. I split the tapes into 1G
chunks to fill the tapes better.

The largest DLE in the weekly sets is about 180GB, that's the reason why
I used 50G tapes and let amanda run 4 tapes. I split the tapes into 10G
chunks to fill the tapes better.

I chose to use 30 tapes * 10GB + (6 tapes * 50GB) * 4 to fit the disk
space of 1,8T.

The keep time of the data does not need to be very long (I do not need
many old versions of specific data). It should be enough to have 2 week
old versions of the backed up data (two runs in WeeklySets and 14 runs
in DailySet).

In my understanding I would need at least L0 dumps every two weeks. To
keep the amount of data small and the time to backup short the amount of
incrementals should be as large as possible.

If you want lvl 0's every two weeks, then should to set the dumpcycle
to two weeks and the runspercycle appropriately.


How do I accomplish all my needs with the resources I have?
Is the way I am thinking completely nonsense?
Are there any best practice rules tipps or recommendations for me?

Beyond I attach my configurations. I removed some default lines, not
needed dumptypes and disktypes and the comments for better readability.
Also I attach the disklists for all my sets.

I hope that anyone can help me

Any comments are appreciated. Thanks in advance!

If you run your backups on the "ragged edge" of sufficient storage,
I would say your (i.e. your mgmt) do not value highly the data
that is being backed up. An extra pair of 1-2TB drives would cost
about $125-$250. Isn't extra safety of your data worth it?

jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)
---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

On Thu, 21 Oct 2010 15:50:34 -0400
Jon LaBadie <jon < at > jgcomp.com> wrote:

As the amount of data is very large (630GB), thus the backup time
would be very long (especially L0's) I divided the data into data
which changes fast and should be backed up daily (DailySet1) and
data which is very static and should be backed up weekly. The
weekly data I also divided into 4 parts (WeeklySet1-4) so that the
runtime of the backup is shorter.

I'm not sure if it is best to make up 5 distinct Amanda configs, or
just one config with customized DLEs and dumptypes. I'd probably try
the latter first. I think it would make the best use of your storage
media rather than targeting specific amounts for each config.

I concur. I'm reluctant to say that a computer is smarter than I am,
but there a few things I am willing to delegate to a computer. Amanda's
backup scheduling is one of them.

I set things up for a 15 day dump cycle, M-F, so three weeks. This
implies *at least* one level 0 per DLE. You will find Amanda staggers
them so as to even out the load. That's much better than one can do
manually, especially as DLE sizes change over time. Also, Amanda will do
more than one level 0 per cycle if it finds the assets to do so.

Be generous with tapes. They are cheaper than your time, and much
cheaper than replacing your data.

--

Charles Curley /"\ ASCII Ribbon Campaign
Looking for fine software \ / Respect for open standards
and/or writing? X No HTML/RTF in email
http://www.charlescurley.com / \ No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0 809C FFF6 4C48 4ECD DFDB

Thank you Jon, Brian and Charles for your answers!

Am 21.10.10 21:50, schrieb Jon LaBadie:
I'm not sure if it is best to make up 5 distinct Amanda configs, or just
one config with customized DLEs and dumptypes. I'd probably try the
latter first. I think it would make the best use of your storage media
rather than targeting specific amounts for each config.

First I started with just one configuration, but the time to backup the
data took more than a day.

My thought was, that the time a backup takes, can be shortened when I
split the data up into parts which should be backuped daily (this data
changes fast) and data that does not change und thus be backuped weekly.

But if I can accomplish that the duration of a backup run takes not that
long and I can backup all of my data on a daily basis I would be very
happy

What do you think about the following considerations:

You don't seem to be using a holding disk so you will be writing
directly to the vtapes. That increases the likelyhood of trashing
something on problems. It also eliminates dumping multiple DLEs
in parallel. Rather one after the next will have to run.

- I have 100G on another partition I can use for a holdingdisk. I will
try to keep the DLEs smaller than 100G to achieve that they fit to the
holdingdisk. I will not be able to fit all of them to the holding disk
size (is this a problem?), but the DLEs which are larger should be
dumped directly to the vtapes, the DLEs which fit will use the holding
disk, right?

- Use a vtape length of 108750 MBytes (= 25 * 4350 (= sizeof a DVD, just
if I would want to archive them to DVDs, but honestly I don't think that
I will do that ever))
tapelength 108750 Mbytes

- Split this tapes into DVD sized parts:
tape_splitsize 4350 mbytes

- dumpcycle 7 days (minimum 1 full dump per week)

- runspercycle 7 (run daily)

- tapecycle 16 (= 16 * 108750 Mbytes (=tapelength) = 1740000 Mbytes
(fits on my tapes-partition (1782415 Mbytes aprox. 1.8T)

- runtapes 2 (needed as the largest DLE is 180G)

If my config is completely stupid what would your configuration be?

Should I use smaller or larger vtapes, thus more or less vtapes and
different splitsizes?
Is the holdingdisk in this configuration usefull?

Even if you want Daily and Weekly to be different configs, rather
than DLEs of one config, consider putting the DLEs from the four
weekly configs into a single config. First benefit you get is
24 tapes of the current size. The dumptype could be set to
"strategy nofull" (or is it "strategy incronly" ?) and use
amadmin "force" commands (from cron) to specify when to do fulls.

It's not that important to know when amanda is doing fulldumps (I think
amanda is much cleverer than me to decide when to do it , but my goal
is to keep backup times and the vtape usage as small as possible and
reasonable.

Will amanda be that "clever" to make the full dumps of each DLE on
different days? If not, is it possible to achieve this?

When I setup this new configuration and run it initially amanda will
make full dumps of all DLEs in the first run, right?

Should I run this first backup manually (I think it'll take more than a
day) and after that set up the cron job?

When using a holding disk and amanda can dump multiple DLEs at a time,
will it be that clever to use DLEs from different hosts to do a bit load
balancing (I do the compressing on client side)? If not, is it possible
to achieve that?

Seriously consider moving to the 3.x versions.

At the moment I would be happy to understand how amanda works and get
amanda running in a way I want it to. I do not want to mess up in a
selfcompiled installation and run into more troubles needed.

If you run your backups on the "ragged edge" of sufficient storage,
I would say your (i.e. your mgmt) do not value highly the data
that is being backed up. An extra pair of 1-2TB drives would cost
about $125-$250. Isn't extra safety of your data worth it?

You are completely right, but there is no management (maybe but my wife
and the problem is not the amount of $ it's the amount of SATA
controllers in my backup "server"

Am 22.10.10 00:48, schrieb Charles Curley:
I set things up for a 15 day dump cycle, M-F, so three weeks. This
implies *at least* one level 0 per DLE. You will find Amanda staggers
them so as to even out the load. That's much better than one can do
manually, especially as DLE sizes change over time. Also, Amanda will do
more than one level 0 per cycle if it finds the assets to do so.

As the time to backup takes more than a day, for me this is not very
good, as more L0s means longer backup time..

Am 21.10.10 22:11, schrieb Brian Cuttler:
The tape savings, the, don't waste a lot of tape on static
files is handled for me by the dump levels as scheduled
by the amanda server. [Discusson how how it does that
withheld for another time, see 'estimate phase']

When I look at my DailySet1 backups for the last days, there are many
backups with the same level. I thought that for example the DLEs
"KonfigFiles" should be backuped once per dumpcycle at L0 and the other
times just incremental as this data _never_ changes. So IMHO it is not
necessary to backup it each day (as it does not change). But obviously I
missed the target here

The problem are not the small DLEs (they do not take very long to
backup), the problem are the larger ones. If the large ones do not
change, amanda will make L0s of them each time and this can take a very
long time each day. Is it possible to force amanda to change that
behaviour (read something about "strategy nofull" / "strategy
incronly")? Does that make sense?

Assuming a DLE is 100G large. The data does not change. I have 10*100G
tapes. I have a dumpcycle of 14 and Amanda choose to make a L0 each day,
because the data does not change. What will happen: Will amanda
overwrite the oldest L0 after 10 days? Or will amanda break and tell me
that it ran out of tapes? This is just hypothetical, maybe it helps me
to understand how amanda works...
What if one day the data changes slightly? Will amanda run a L1 in the
next run? What will amanda do after the L1?

Btw. does in my amoverview DLE hochschw RootHome mean, that amanda made
on 20.-22.10. always incrementals or are incrementals only made when
there is a switch from 0 to 1 and 1 to 2 and so on, so that amanda made
fulls on 19., then made an incremental on 20. and again fulls on 21. and
22. What happened on 23.? Has the dumpcycle been over and amanda
switched back to L0 and made a full again?

root < at > store02:~# su backup -c "amoverview DailySet1 -skipmissed"

date 10 10 10 10 10 10 10 10
host disk 18 19 20 21 22 23 24 25

calendar Kalender 0 0 0 0 0 0 0
calendar KonfigFiles 0 0 0 0 0 0 0
calendar Roothome 0 0 0 0 0 0 0
hochschw Backups 0 0 0 0 0 0 0
hochschw KonfigFiles 0 0 0 0 0 0 0
hochschw Mailserver 0 0 0 0 1 1 1
hochschw RootHome 0 1 1 1 0 0 0
riegerin KonfigFiles 0 0 0 0 0 0 0
riegerin RootHome 0 0 0 0 0 0 0
store01. Dokumente 11 11 11 11 11 11 11 11
store01. KonfigFiles 0 0 0 0 0 0 0
store01. RootHome 0 0 0 0 0 0 0
store01. Scratch 1 1 1 1 0 00 00
store01. WindowsHomes 11 11 11 11 11 11 11 11
store02. KonfigFiles 0 0 0 0 0 0 0
store02. RootHome 0 0 0 0 0 0 0

Thank you very much for your help!

Cheers,
Thomas

Thomas,

Regarding work-area and tapes...

Amanda gets its parallelism by running multiple dumpers
simultaniously, multiple DLEs being dumped concurrently
to the work area, as each DLE completes it is flushed
to disk (excepting of course if you are using tape tuning
parameters but we will disregard them and stick to classic
behavor for now).

If you are dumping directly to tape or vtape, because you
lack a work area you are forcing amanda to run the dumps
in a non-parallelized mannor, esentially as would occur
if you where simply running manual TAR or DUMP commands
one after the other (except of course that amanda is setting
the dump levels for you).

If you have a 'sufficient' work area you will run multiple
dumps and you will see in your amanda reports that dump time
will be some multiple of the run or wall clock time. Currently
I imaging those values are equal 1:1.

You will also see that the time to dump each partition and the
time to tape each partition will no long be the same, since
you can push a work-area file to tape (or vtape) a lot more
quickly than you can dump the file to the work area.

This is where your real savings will be, time-wise.

The work area will not be used if the DLE is larger than the
dump area, if you have room for more than one dump in the
work area you will begin to have your parallelism.

I DO NOT KNOW, if the new version of amanda which can write
multiple tapes at the same time will allow multiple dumps
directly to vtapeS. Still, even if it does it would result
is an effective dump limit of the number of vtapes you are
willing to commit to each amanda run.

As far as DLE space and work area, any specific DLE is restricted
to total work area space, if you are using "chunksize", ie that
is the DLE is written not as a single file to the work area but
as multiple (perhaps numerous) smaller files (I set my chunksize
as 1 Gig) allowing any large DLE to span multiple work areas.

As long as you have room to pre-allocate to a DLE based on its
size estimate you can allow amanda to start another dump thread.

Hope this helps...

Brian



On Mon, Oct 25, 2010 at 10:26:42AM +0200, Thomas Marko wrote:
Thank you Jon, Brian and Charles for your answers!

Am 21.10.10 21:50, schrieb Jon LaBadie:
I'm not sure if it is best to make up 5 distinct Amanda configs, or just
one config with customized DLEs and dumptypes. I'd probably try the
latter first. I think it would make the best use of your storage media
rather than targeting specific amounts for each config.

First I started with just one configuration, but the time to backup the
data took more than a day.

My thought was, that the time a backup takes, can be shortened when I
split the data up into parts which should be backuped daily (this data
changes fast) and data that does not change und thus be backuped weekly.

But if I can accomplish that the duration of a backup run takes not that
long and I can backup all of my data on a daily basis I would be very
happy

What do you think about the following considerations:

You don't seem to be using a holding disk so you will be writing
directly to the vtapes. That increases the likelyhood of trashing
something on problems. It also eliminates dumping multiple DLEs
in parallel. Rather one after the next will have to run.

- I have 100G on another partition I can use for a holdingdisk. I will
try to keep the DLEs smaller than 100G to achieve that they fit to the
holdingdisk. I will not be able to fit all of them to the holding disk
size (is this a problem?), but the DLEs which are larger should be
dumped directly to the vtapes, the DLEs which fit will use the holding
disk, right?

- Use a vtape length of 108750 MBytes (= 25 * 4350 (= sizeof a DVD, just
if I would want to archive them to DVDs, but honestly I don't think that
I will do that ever))
tapelength 108750 Mbytes

- Split this tapes into DVD sized parts:
tape_splitsize 4350 mbytes

- dumpcycle 7 days (minimum 1 full dump per week)

- runspercycle 7 (run daily)

- tapecycle 16 (= 16 * 108750 Mbytes (=tapelength) = 1740000 Mbytes
(fits on my tapes-partition (1782415 Mbytes aprox. 1.8T)

- runtapes 2 (needed as the largest DLE is 180G)

If my config is completely stupid what would your configuration be?

Should I use smaller or larger vtapes, thus more or less vtapes and
different splitsizes?
Is the holdingdisk in this configuration usefull?

Even if you want Daily and Weekly to be different configs, rather
than DLEs of one config, consider putting the DLEs from the four
weekly configs into a single config. First benefit you get is
24 tapes of the current size. The dumptype could be set to
"strategy nofull" (or is it "strategy incronly" ?) and use
amadmin "force" commands (from cron) to specify when to do fulls.

It's not that important to know when amanda is doing fulldumps (I think
amanda is much cleverer than me to decide when to do it , but my goal
is to keep backup times and the vtape usage as small as possible and
reasonable.

Will amanda be that "clever" to make the full dumps of each DLE on
different days? If not, is it possible to achieve this?

When I setup this new configuration and run it initially amanda will
make full dumps of all DLEs in the first run, right?

Should I run this first backup manually (I think it'll take more than a
day) and after that set up the cron job?

When using a holding disk and amanda can dump multiple DLEs at a time,
will it be that clever to use DLEs from different hosts to do a bit load
balancing (I do the compressing on client side)? If not, is it possible
to achieve that?

Seriously consider moving to the 3.x versions.

At the moment I would be happy to understand how amanda works and get
amanda running in a way I want it to. I do not want to mess up in a
selfcompiled installation and run into more troubles needed.

If you run your backups on the "ragged edge" of sufficient storage,
I would say your (i.e. your mgmt) do not value highly the data
that is being backed up. An extra pair of 1-2TB drives would cost
about $125-$250. Isn't extra safety of your data worth it?

You are completely right, but there is no management (maybe but my wife
and the problem is not the amount of $ it's the amount of SATA
controllers in my backup "server"

Am 22.10.10 00:48, schrieb Charles Curley:
I set things up for a 15 day dump cycle, M-F, so three weeks. This
implies *at least* one level 0 per DLE. You will find Amanda staggers
them so as to even out the load. That's much better than one can do
manually, especially as DLE sizes change over time. Also, Amanda will do
more than one level 0 per cycle if it finds the assets to do so.

As the time to backup takes more than a day, for me this is not very
good, as more L0s means longer backup time..

Am 21.10.10 22:11, schrieb Brian Cuttler:
The tape savings, the, don't waste a lot of tape on static
files is handled for me by the dump levels as scheduled
by the amanda server. [Discusson how how it does that
withheld for another time, see 'estimate phase']

When I look at my DailySet1 backups for the last days, there are many
backups with the same level. I thought that for example the DLEs
"KonfigFiles" should be backuped once per dumpcycle at L0 and the other
times just incremental as this data _never_ changes. So IMHO it is not
necessary to backup it each day (as it does not change). But obviously I
missed the target here

The problem are not the small DLEs (they do not take very long to
backup), the problem are the larger ones. If the large ones do not
change, amanda will make L0s of them each time and this can take a very
long time each day. Is it possible to force amanda to change that
behaviour (read something about "strategy nofull" / "strategy
incronly")? Does that make sense?

Assuming a DLE is 100G large. The data does not change. I have 10*100G
tapes. I have a dumpcycle of 14 and Amanda choose to make a L0 each day,
because the data does not change. What will happen: Will amanda
overwrite the oldest L0 after 10 days? Or will amanda break and tell me
that it ran out of tapes? This is just hypothetical, maybe it helps me
to understand how amanda works...
What if one day the data changes slightly? Will amanda run a L1 in the
next run? What will amanda do after the L1?

Btw. does in my amoverview DLE hochschw RootHome mean, that amanda made
on 20.-22.10. always incrementals or are incrementals only made when
there is a switch from 0 to 1 and 1 to 2 and so on, so that amanda made
fulls on 19., then made an incremental on 20. and again fulls on 21. and
22. What happened on 23.? Has the dumpcycle been over and amanda
switched back to L0 and made a full again?

root < at > store02:~# su backup -c "amoverview DailySet1 -skipmissed"

date 10 10 10 10 10 10 10 10
host disk 18 19 20 21 22 23 24 25

calendar Kalender 0 0 0 0 0 0 0
calendar KonfigFiles 0 0 0 0 0 0 0
calendar Roothome 0 0 0 0 0 0 0
hochschw Backups 0 0 0 0 0 0 0
hochschw KonfigFiles 0 0 0 0 0 0 0
hochschw Mailserver 0 0 0 0 1 1 1
hochschw RootHome 0 1 1 1 0 0 0
riegerin KonfigFiles 0 0 0 0 0 0 0
riegerin RootHome 0 0 0 0 0 0 0
store01. Dokumente 11 11 11 11 11 11 11 11
store01. KonfigFiles 0 0 0 0 0 0 0
store01. RootHome 0 0 0 0 0 0 0
store01. Scratch 1 1 1 1 0 00 00
store01. WindowsHomes 11 11 11 11 11 11 11 11
store02. KonfigFiles 0 0 0 0 0 0 0
store02. RootHome 0 0 0 0 0 0 0

Thank you very much for your help!

Cheers,
Thomas
---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

On Mon, Oct 25, 2010 at 10:26:42AM +0200, Thomas Marko wrote:
Thank you Jon, Brian and Charles for your answers!


Just a couple of thoughts ...

Am 21.10.10 21:50, schrieb Jon LaBadie:

Even if you want Daily and Weekly to be different configs, rather
than DLEs of one config, consider putting the DLEs from the four
weekly configs into a single config. First benefit you get is
24 tapes of the current size. The dumptype could be set to
"strategy nofull" (or is it "strategy incronly" ?) and use
amadmin "force" commands (from cron) to specify when to do fulls.

It's not that important to know when amanda is doing fulldumps (I think
amanda is much cleverer than me to decide when to do it , but my goal
is to keep backup times and the vtape usage as small as possible and
reasonable.

Will amanda be that "clever" to make the full dumps of each DLE on
different days? If not, is it possible to achieve this?

Amanda tries to get balance things so that each days dump will
backup the same amount of data (actually, if compressed, the
same amount of "tape" is more accurate). But it can't get there
until it has a history of dumps and compression rates. Even
with a history, if you have a relatively small number of DLEs
and very different sizes, it can never completely balance
things. Under these conditions amanda seems to full-dump the
small DLEs frequently.


When I setup this new configuration and run it initially amanda will
make full dumps of all DLEs in the first run, right?

Should I run this first backup manually (I think it'll take more than a
day) and after that set up the cron job?

I recently reworked my backup setup, essentially started over.
At first I started with small pieces. If you are configured to use
timestamps (usetimestamps yes) you can run multiple amdumps in one
day. So I commented out all the DLEs in the disklist file and
gradually uncommented them over several dumps. First a server DLE.
Next another server DLE and one or two client DLEs. Third one
more DLE from each host. Then for each regularly schedulded dump
I uncommented one or two additional DLEs from each host. In my
case, this took about 3 days. I suspect others would suggest
dumping everything right from the start.

When using a holding disk and amanda can dump multiple DLEs at a time,
will it be that clever to use DLEs from different hosts to do a bit load
balancing (I do the compressing on client side)? If not, is it possible
to achieve that?

There are parameters that control how many total simultaneous dumps max
and how many simultaneous from one client max. I think I have them set
to 4 and 1.


Seriously consider moving to the 3.x versions.

At the moment I would be happy to understand how amanda works and get
amanda running in a way I want it to. I do not want to mess up in a
selfcompiled installation and run into more troubles needed.


Are prebuilt binaries available for your OSs at zmanda.com?

If you run your backups on the "ragged edge" of sufficient storage,
I would say your (i.e. your mgmt) do not value highly the data
that is being backed up. An extra pair of 1-2TB drives would cost
about $125-$250. Isn't extra safety of your data worth it?

You are completely right, but there is no management (maybe but my wife
and the problem is not the amount of $ it's the amount of SATA
controllers in my backup "server"

Same as my setup. And I ran into the same situation, only one channel
of SATA on my selected amanda server (a very old Pentium-3). I just
spent $29 more for another controller so I could add two more disks.
One isn't even in a cage, it is lying on the bottom of the case.

I also moved from the previous amanda server, two external USB drives
and the USB 2.0 interface card I previously purchased for the old host.
Neither had USB 2 when purchased.

jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

Jon,
Thomas,

On Mon, Oct 25, 2010 at 12:31:47PM -0400, Jon LaBadie wrote:

When I setup this new configuration and run it initially amanda will
make full dumps of all DLEs in the first run, right?

Should I run this first backup manually (I think it'll take more than a
day) and after that set up the cron job?

I recently reworked my backup setup, essentially started over.
At first I started with small pieces. If you are configured to use
timestamps (usetimestamps yes) you can run multiple amdumps in one
day. So I commented out all the DLEs in the disklist file and
gradually uncommented them over several dumps. First a server DLE.
Next another server DLE and one or two client DLEs. Third one
more DLE from each host. Then for each regularly schedulded dump
I uncommented one or two additional DLEs from each host. In my
case, this took about 3 days. I suspect others would suggest
dumping everything right from the start.


I have used both aproaches, largely depending on how long I
think the dump will take and how much of the tape I think we
are likely to consume.

I'm also more likely to run a complete dump initially for 'new'
services where the server is just being put online and there is
still data/services being migrated to it. As when we started to
migrate all of the user home directories to a new platform and
the old platform was still in production.

In Thomas' case I'd be more likely to stage things as Jon suggested
as we already know you are pushing limits on tape capacity and the
work area has not yet been implemented.

---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

Brian,

thank you for your patience. As I am pretty new to the "backup business"
it's not easy to understand how some things work.

Am 25.10.10 16:25, schrieb Brian Cuttler:

Amanda gets its parallelism by running multiple dumpers
simultaniously, multiple DLEs being dumped concurrently
to the work area, as each DLE completes it is flushed
to disk (excepting of course if you are using tape tuning
parameters but we will disregard them and stick to classic
behavor for now).

As I understand this, a work area should be as large as a dump can be in
a run. So the size of a specific DLE is not that important than the size
of the whole dump (which depends on the levels of the dumps of the DLEs
to be run on that day), means the sum of all data to be dumped to the
tapes on that day, right?

If you have a 'sufficient' work area you will run multiple
dumps and you will see in your amanda reports that dump time
will be some multiple of the run or wall clock time. Currently
I imaging those values are equal 1:1.

What is 'sufficient'? Is it the amount of space all DLEs take? Or just
the amount of space a run of amdump takes? How can I calculate the
amount of space a dump takes, as I do not know when and which DLE is
full dumped and which DLEs are incrementals?

Also the question of dimensioning the tapelength is not really clear for
me. Should I go for tapes with the length of the largest DLE (+ some GB
spare)? Or should the tapes be smaller and should I use tape_splitsize
and runtapes as mentioned in my last post?

At the moment my vtapes reside on a RAID5 (linux software raid). Do you
think for that purpose it is sufficient to use a RAID0? The pro would be
way more space and faster writes with the con of way higher risk to
loose the data. But if I have real tapes there could also be a tape lost
or destroyed... Another possibilty could be to do not use a RAID and to
allocate the tapes across three different disks (partitions). Then I
will not loose all vtapes at once.
Hmmm backup is a very complicate part of IT business

Thank you for your help!

/Thomas

Jon,

thank you for help!

Am 25.10.10 18:31, schrieb Jon LaBadie:
Amanda tries to get balance things so that each days dump will
backup the same amount of data (actually, if compressed, the
same amount of "tape" is more accurate). But it can't get there
until it has a history of dumps and compression rates. Even
with a history, if you have a relatively small number of DLEs
and very different sizes, it can never completely balance
things. Under these conditions amanda seems to full-dump the
small DLEs frequently.

So the best strategy would be to "balance" the DLEs to have the same
size. The issue that small DLEs are always full dumped is not that
problem, because we are talking about some Megs /etc and some files in
/root... The problem are the big ones. Should I split the big DLEs up
into handsome parts, so that most of them are nearly the same size?
This also leads me to the question regarding the dimensions of
tape_length and splitsize and so on a asked Brian in a posting above:

"Should I go for tapes with the length of the largest DLE (+ some GB
spare)? Or should the tapes be smaller and should I use tape_splitsize
and runtapes as mentioned in my last post?"

There are parameters that control how many total simultaneous dumps max
and how many simultaneous from one client max. I think I have them set
to 4 and 1.

You are talking about "inparallel" and "maxdumps", right?

Same as my setup. And I ran into the same situation, only one channel
of SATA on my selected amanda server (a very old Pentium-3). I just
spent $29 more for another controller so I could add two more disks.
One isn't even in a cage, it is lying on the bottom of the case.

Do you run a RAID over the disks? If yes, which level? If no, do you
allocate the vtapes over all disks?

Thank you very much!

/Thomas

On Wed, Oct 27, 2010 at 09:42:34AM +0200, Thomas Marko wrote:
Jon,

thank you for help!

Am 25.10.10 18:31, schrieb Jon LaBadie:
Amanda tries to get balance things so that each days dump will
backup the same amount of data (actually, if compressed, the
same amount of "tape" is more accurate). But it can't get there
until it has a history of dumps and compression rates. Even
with a history, if you have a relatively small number of DLEs
and very different sizes, it can never completely balance
things. Under these conditions amanda seems to full-dump the
small DLEs frequently.

So the best strategy would be to "balance" the DLEs to have the same
size.

Rather than saying it is the "best strategy" I'd say the best
situation is all DLEs the same size. But we live in the real
world (most of the time and that is not a reasonable expectation.

The issue that small DLEs are always full dumped is not that
problem, because we are talking about some Megs /etc and some files in
/root... The problem are the big ones.

Right, you spend some time on the big problems and let amanda
deal with the small ones.

Should I split the big DLEs up
into handsome parts, so that most of them are nearly the same size?

Yes, but don't go overboard trying to get them exact. And sometimes
it is impossible to split them up. An example is I have my Windows
"machines" are virtual. Their "virtual filesystem" is a single
file on its own filesystem. I have no way of spliting it. But if
you have a large /home, it is easy to split it up based on the
leading character of the user names. This has been covered several
times on the list and may be in the wiki.

This also leads me to the question regarding the dimensions of
tape_length and splitsize and so on a asked Brian in a posting above:

"Should I go for tapes with the length of the largest DLE (+ some GB
spare)? Or should the tapes be smaller and should I use tape_splitsize
and runtapes as mentioned in my last post?"

Tape splitting has addressed a lot of old amanda "problems". I'd
recommend using it. The only negative is recovery of dumps without
amanda software is slightly more difficult. Normal recovery with
amanda software is still fine.


There are parameters that control how many total simultaneous dumps max
and how many simultaneous from one client max. I think I have them set
to 4 and 1.

You are talking about "inparallel" and "maxdumps", right?

Same as my setup. And I ran into the same situation, only one channel
of SATA on my selected amanda server (a very old Pentium-3). I just
spent $29 more for another controller so I could add two more disks.
One isn't even in a cage, it is lying on the bottom of the case.

Do you run a RAID over the disks? If yes, which level? If no, do you
allocate the vtapes over all disks?

No raid, but I do have multiple disks collected under linux' LVM into
single filesystems. One is the 2 x 1TB disks mentioned above and
the second is a pair of 300GB external USB drives. I was thinking
of including the USB drives and internal drives into a single volume
group, but I was not confident the slow start-up time of the USB
drives would not upset things. So I keep them is a separate VG.

Total formatted capacity of the two VGs is 2.4TB. I divided this
into 100 25GB vtapes. That is greater than the capacity, but I'm
counting on the last tape of each run being only partially full.
My experience says that is reasonable as the two VGs run between
80 and 90% full.

Just as a single datapoint, my environment is 7 clients, 27 DLEs,
250GB of data currently, largest DLE is 37GB (24GB compressed).
I run a dumpcycle of 2 wks, daily amdump runs. Runtapes is 4
but typically only 2, sometimes 3 are used each run. Currently
37 days of dumps are on the 100 vtapes, about 2.5 dumpcycles.

HTH
jl
--
Jon H. LaBadie jon < at > jgcomp.com
JG Computing
12027 Creekbend Drive (703) 787-0884
Reston, VA 20194 (703) 787-0922 (fax)

Hi Jon,

thank you very much for your help! I rebuilt my whole configuration. Now
I use only one Backup Set (DailySet1) with several DLEs I tried to keep
from similar size.

Just for the information of people reading this thread in future my
chosen configuration:

- Two HDDs each 320GB (in vendor calculation) in one LVM VG with
striping (for better performance) used as holding disk gives me 587 GB
free space

- Three HDDs each 1T (in vendor calculation) in one LVM VG without
striping (for better data assurance) used as space for vtapes gives me
2,7 TB free space

- dumpcycle 2 weeks
- runspercycle 14
- tapecycle 108 tapes (each 25GB length)
- tape_splitsize 1GB

For my initial backup I set runtapes to 108 to be sure that all of the
data can be fully backuped. I decided to run amdump initially for all
DLEs and not one by one as I think I will get a better tape usage. I
want to change it to 4 (* tape-length, which is more than the size of my
largest DLE) after I have a L0 backup of all DLEs.

I ran amdump nearly without problems . Most of the DLEs where
backuped, but some not. These problems I will ask for help in another
thread as it is OT here.

Thanks again to you and also Brian and Charles helping me to understand
how Amanda works!

Cheers,
Thomas