Still unable to resume a partial backup

Hello,

the issue I've raised 4 years ago:

https://sourceforge.net/p/backuppc/mailman/message/29727529/

still persists with BackupPC 3.3.0. Basically what happens is that
a combination of large new files and a slow connection between
backup server and client means that a full backup eventually times
out. Before that, it doesn't seem like data is being shoved across
the ether, so the ClientTimeout action seems legit. However, the
backup will be far from finished, and eventually be saved as
a partial.

The next day, BackupPC awakes, tries to resume, and eventually times
out again.

And then this happens every subsequent day.

Instead, I'd expect BackupPC to keep growing the partial backup so
that it eventually becomes a full backup.

I am using rsync+ssh.

Here are relevant configuration items:

$Conf{ClientTimeout} = 72000;
$Conf{PartialAgeMax} = 3;

Any tips appreciated…

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

spooning leads to forking
-- seen on a t-shirt by david & goliath

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 4/08/2016 22:17, martin f krafft wrote:
[quote]Hello,

the issue I've raised 4 years ago:

https://sourceforge.net/p/backuppc/mailman/message/29727529/

still persists with BackupPC 3.3.0. Basically what happens is that
a combination of large new files and a slow connection between
backup server and client means that a full backup eventually times
out. Before that, it doesn't seem like data is being shoved across
the ether, so the ClientTimeout action seems legit. However, the
backup will be far from finished, and eventually be saved as
a partial.

The next day, BackupPC awakes, tries to resume, and eventually times
out again.

And then this happens every subsequent day.

Instead, I'd expect BackupPC to keep growing the partial backup so
that it eventually becomes a full backup.

I am using rsync+ssh.

Here are relevant configuration items:

$Conf{ClientTimeout} = 72000;
$Conf{PartialAgeMax} = 3;

Any tips appreciated…

[/quote]I forget the details but a couple of things that can help:
1) Ensure you enable SSH keepalives to keep your NAT firewall open
2) You can look to split remote files before the backup, and exclude the
original large files (sometimes this is helpful anyway, because some of
the chunks will not change, I do this with sql server dumps, ensure you
don't compress them though, or else every chunk will 100% change).
3) Ensure that you can backup any file within the ClientTimeout, so if
the largest file you will backup is 2.5GB, and you have a 2Mbps line
(ie, 2M up at the client and 2M down at the server) then ClientTimeout
will need to be 2500 * 8 / 2 = 10,000 seconds * 1.25 (never get 100%) =
12,500 seconds. You will need to calculate for your own maximum size
file and bandwidth. I'd also suggest doubling this again to 25,000 in
the example.

Remember, ClientTimeout is supposed to be a worst case scenario if the
connection actually hangs. If your firewall/nat router/ssh will tear
down the tunnel properly, causing rsync to fail, then you will never hit
the ClientTimeout, so you can safely make this a much bigger number.

The problem is backuppc will not accept a partial file, either the full
file is received/saved in the partial backup, or none of the file is
saved. I don't expect this will change.

Regards,
Adam

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Hey Adam, thanks for your quick response. I have a few points to
add:

[quote]1) Ensure you enable SSH keepalives to keep your NAT firewall open
[/quote]
Yes, of course these are enabled.

[quote]2) You can look to split remote files before the backup, and exclude the
original large files (sometimes this is helpful anyway, because some of
the chunks will not change, I do this with sql server dumps, ensure you
don't compress them though, or else every chunk will 100% change)
[/quote]
This is unfortunately not a scalable solution (just like excluding
some files to let the backups complete stepwise isn't either).

[quote]3) Ensure that you can backup any file within the ClientTimeout,
[/quote]
Is this necessary? Isn't ClientTimeout about killing the connection
after a period of time without any traffic?

[quote]The problem is backuppc will not accept a partial file, either the
full file is received/saved in the partial backup, or none of the
file is saved. I don't expect this will change.
[/quote]
I don't think it can change. But if I have three huge files, and the
backup always dies half way through the second, then I would hope
that BackupPC learnt to reuse the first, already transferred file
instead of repeating the same thing as the day before and failing
half way through the second file again… I think this is the core of
the problem, somehow… it does not seem like partial backups get
updated on resume…

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

"da haben wir es also: eine kirchliche ordnung mit priesterschaft,
theologie, kultus, sakrament;
kurz, alles das, was jesus von nazareth bekämpft hatte..."
- friedrich nietzsche

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 4/08/2016 23:43, martin f krafft wrote:
[quote]3) Ensure that you can backup any file within the ClientTimeout,
Is this necessary? Isn't ClientTimeout about killing the connection
after a period of time without any traffic?
[/quote]Almost, but the timer is only updated after each file has been
transferred, not after each chunk/byte of a file.
[quote][quote]The problem is backuppc will not accept a partial file, either the
full file is received/saved in the partial backup, or none of the
file is saved. I don't expect this will change.
[/quote]I don't think it can change. But if I have three huge files, and the
backup always dies half way through the second, then I would hope
that BackupPC learnt to reuse the first, already transferred file
instead of repeating the same thing as the day before and failing
half way through the second file again… I think this is the core of
the problem, somehow… it does not seem like partial backups get
updated on resume…

[/quote]It should work as you said, but if you never have enough time to
transfer the second file, then you won't actually proceed. BackupPC will
still check every file "before" the second file in case there have been
changes there, but ultimately, if the second file is too big to transfer
within the allotted time, then it can't succeed.

Hope that helps.

Regards,
Adam

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-04 15]:
[quote]On 4/08/2016 23:43, martin f krafft wrote:
[quote]3) Ensure that you can backup any file within the ClientTimeout,
Is this necessary? Isn't ClientTimeout about killing the connection
after a period of time without any traffic?
[/quote]Almost, but the timer is only updated after each file has been
transferred, not after each chunk/byte of a file.
[/quote]
Oh, that actually explains *a lot*. Thanks. I'll just push the value
to two weeks now ;)

[quote]It should work as you said, but if you never have enough time to
transfer the second file, then you won't actually proceed.
BackupPC will still check every file "before" the second file in
case there have been changes there, but ultimately, if the second
file is too big to transfer within the allotted time, then it
can't succeed.
[/quote]
I do wonder if it woulnd't make sense to

(a) randomise the order of files
(b) update partial backups

for the combination of those two will mean that over time, even
a partial backup will become more and more useful, don't you think?

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

"a fundamentalist is someone who hates sin more than he loves virtue."
-- john h. schaar

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/08/16 04:42, martin f krafft wrote:
[quote]also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-04 15]:
[quote]On 4/08/2016 23:43, martin f krafft wrote:
[/quote]
[quote]It should work as you said, but if you never have enough time to
transfer the second file, then you won't actually proceed.
BackupPC will still check every file "before" the second file in
case there have been changes there, but ultimately, if the second
file is too big to transfer within the allotted time, then it
can't succeed.
[/quote]I do wonder if it woulnd't make sense to

(a) randomise the order of files
(b) update partial backups

for the combination of those two will mean that over time, even
a partial backup will become more and more useful, don't you think?

[/quote]The order that files are processed depends on the client, without using
non-standard client tools, we can't influence/change that.
We do update a partial backup, as long as the new partial contains more
files than the previous partial, but it doesn't even save a partially
transferred file. In some ways, (I've also asked for this a few years
ago), it would be nice, because as you said, over time you would get
more and more of the file, and eventually complete it (and each time it
would be "more useful"). However, the decision was made to ensure that
either the file is correct (complete) or missing, and that argument does
also make sense (I see the reasons for both options, just neither option
is 100% right for everybody).

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-05 01]:
[quote]We do update a partial backup, as long as the new partial contains
more files than the previous partial, but it doesn't even save
a partially transferred file.
[/quote]
I think this is fine, and a partially transferred file is actually
possibly less useful than not having it, and if only it gets your
hopes up, or you don't notice immediately. Gosh, imagine watching
Alien from backup and not being able to see the end! ;)

In my case, there aren't huge files (or rather, I try to handle them
differently (git-annex)). So the partial file issue is none for me.

Yet, backups still sometimes never complete. I think it must be
a combination of the slow link, and the fact that (I think) BackupPC
transfers too much data for files that have already been backed up.
I've started another thread on this and will continue there.

https://adsm.org/lists/html/BackupPC-users/2016-08/msg00018.html

Thanks, Adam and everyone!

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

"'oh, that was easy,' says Man, and for an encore goes on to prove
that black is white and gets himself killed on the next zebra
crossing."
-- douglas adams, "the hitchhiker's guide to the galaxy"

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/