How do you monitor a backup process?

Hey,

some of the backup processes here run for hours, and there are often
reasons why I want to check on what's going on.

How do you monitor backups in real-time? XferLOG.z can't be tail'd,
and attaching strace or lsof to the running processes just isn't
very sexy.

Can you fathom a good method by which I can keep a good eye on
what's going on, e.g. a way to have BackupPC write to the host's log
file something like

2016-08-04 14:39:11 considering file: /foo/bar/baz (cpool:a/b/c/defgh…)
2016-08-04 14:39:12 skipping file: /foo/bar/baz — already in pool
2016-08-04 14:39:13 transferring file: /foo/bar/baz
2016-08-04 14:39:17 done transferring: /foo/bar/baz

I've tried adding --verbose and --progress to rsync, I've tried
increasing the XferLogLevel beyond 1 (but the XferLOG isn't
real-time in the way I'd seek it).

Maybe one solution would be to write the XferLOG without zlib
compression and only compress it later? Is this somehow
configurable?

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

"if a man treats life artistically, his brain is his heart."
-- oscar wilde

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 4/08/2016 22:59, martin f krafft wrote:
[quote]Hey,

some of the backup processes here run for hours, and there are often
reasons why I want to check on what's going on.

How do you monitor backups in real-time? XferLOG.z can't be tail'd,
and attaching strace or lsof to the running processes just isn't
very sexy.

Can you fathom a good method by which I can keep a good eye on
what's going on, e.g. a way to have BackupPC write to the host's log
file something like

[/quote]I've used ls -l /proc/pid/fds or strace or lsof etc... all work, some
are better on the client rather than the backuppc server.
I've also used tail -f XferLOG | Backuppc_zcat which does work, but
doesn't update in real time (ie, you have to wait for a number of lines
of log output before you see the update.....

Not sure of a "better" way.... Backuppc 4.0 includes a counter for
number of files xfered though that doesn't help for BPC 3.x

Regards,
Adam

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

You can follow XferLog with

/usr/share/BackupPC/bin/BackupPC_zcat <pathto>/XferLOG.z | tail

but it is buffered, so not completely current
Also on the backup host, you can get the  process id of the current dump processes (there will be two per host during file transfer), and do
(sudo) ls -l /proc/{pid1,pid2}/fd
if a file is being written to backup it will show up in this list. But be aware that there are times (sometimes long) when files are not being written

On Thu, Aug 4, 2016 at 8:59 AM, martin f krafft <madduck < at > madduck.net ([email]madduck < at > madduck.net[/email])> wrote:
[quote]Hey,

some of the backup processes here run for hours, and there are often
reasons why I want to check on what&#39;s going on.

How do you monitor backups in real-time? XferLOG.z can&#39;t be tail&#39;d,
and attaching strace or lsof to the running processes just isn&#39;t
very sexy.

Can you fathom a good method by which I can keep a good eye on
what&#39;s going on, e.g. a way to have BackupPC write to the host&#39;s log
file something like

  2016-08-04 14:39:11 considering file: /foo/bar/baz (cpool:a/b/c/defgh…)
  2016-08-04 14:39:12 skipping file: /foo/bar/baz — already in pool
  2016-08-04 14:39:13 transferring file: /foo/bar/baz
  2016-08-04 14:39:17 done transferring: /foo/bar/baz

I&#39;ve tried adding --verbose and --progress to rsync, I&#39;ve tried
increasing the XferLogLevel beyond 1 (but the XferLOG isn&#39;t
real-time in the way I&#39;d seek it).

Maybe one solution would be to write the XferLOG without zlib
compression and only compress it later? Is this somehow
configurable?

--
< at > martinkrafft | [url=http://madduck.net/]http://madduck.net/[/url] | [url=http://two.sentenc.es/]http://two.sentenc.es/[/url]

"if a man treats life artistically, his brain is his heart."
                                                        -- oscar wilde

spamtraps: madduck.bogus < at > madduck.net ([email]madduck.bogus < at > madduck.net[/email])

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net ([email]BackupPC-users < at > lists.sourceforge.net[/email])
List:    [url=https://lists.sourceforge.net/lists/listinfo/backuppc-users]https://lists.sourceforge.net/lists/listinfo/backuppc-users[/url]
Wiki:    [url=http://backuppc.wiki.sourceforge.net]http://backuppc.wiki.sourceforge.net[/url]
Project: [url=http://backuppc.sourceforge.net/]http://backuppc.sourceforge.net/[/url]

[/quote]

On 04/08/2016 14:59, martin f krafft wrote:
[quote]some of the backup processes here run for hours, and there are often
reasons why I want to check on what's going on.
[/quote]
In the past, I've used things similar to what's mentioned at these
pages:
http://sysadminnotebook.blogspot.nl/2011/09/watch-backuppc-progress.html
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/backuppc-21/backuppc-progress-109678/?sid=a61a27a3e619e76f4f0348ff8f72c9ce

Best,
Koen

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-04 16]:
[quote]I've used ls -l /proc/pid/fds or strace or lsof etc... all work,
some are better on the client rather than the backuppc server.
[/quote]
In fact, I found none of those useful on the server.

[quote]I've also used tail -f XferLOG | Backuppc_zcat which does work,
but doesn't update in real time (ie, you have to wait for a number
of lines of log output before you see the update.....
[/quote]
I've tried this, but I get:

/usr/share/backuppc/bin/BackupPC_zcat: can't uncompress stdin

This is using BackupPC 3.3.0 (Debian stable)

[quote]Not sure of a "better" way.... Backuppc 4.0 includes a counter for
number of files xfered though that doesn't help for BPC 3.x
[/quote]
The counter isn't really that useful, I think, especially not if it
doesn't have a "X of Y files" total that doesn't change (cf. rsync,
which is kinda useless, as the total keeps increasing).

The more I think about it, the more I want XferLOG
uncompressed/unbuffered, but also structured in a way so that it
starts a new line when it inspects a file, and then finishes the
line with details and the verdict (same, create, link, …)

also sprach Tony Schreiner <anthony.schreiner < at > bc.edu> [2016-08-04 15]:
[quote]Also on the backup host, you can get the process id of the current dump
processes (there will be two per host during file transfer), and do

(sudo) ls -l /proc/{pid1,pid2}/fd

if a file is being written to backup it will show up in this list. But be
aware that there are times (sometimes long) when files are not being written
[/quote]
What happens during those times?

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

"zum christentum wird man nicht geboren,
man muß dazu nur krank genug sein."
- friedrich nietzsche

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

On 05/08/16 04:52, martin f krafft wrote:
[quote]also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-04 16]:
[quote]I've used ls -l /proc/pid/fds or strace or lsof etc... all work,
some are better on the client rather than the backuppc server.
[/quote]In fact, I found none of those useful on the server.

[quote]I've also used tail -f XferLOG | Backuppc_zcat which does work,
but doesn't update in real time (ie, you have to wait for a number
of lines of log output before you see the update.....
[/quote]I've tried this, but I get:

/usr/share/backuppc/bin/BackupPC_zcat: can't uncompress stdin

This is using BackupPC 3.3.0 (Debian stable)
[/quote]
Sorry, I've not used BPC 3.x in years...
Maybe try this:
tail -f -n +0 blah.log | /usr/share/backuppc/bin/BackupPC_zcat -
You need to include the beginning of the file or else it won't detect
the compression header.... Also, the - frequently means use stdin when a
filename parameter is required, it may or may not be needed.
[quote]
[quote]Not sure of a "better" way.... Backuppc 4.0 includes a counter for
number of files xfered though that doesn't help for BPC 3.x
[/quote]The counter isn't really that useful, I think, especially not if it
doesn't have a "X of Y files" total that doesn't change (cf. rsync,
which is kinda useless, as the total keeps increasing).
[/quote]It includes the total number of files from the previous backup... so
generally it is pretty useful (unless the client has added a huge number
of files in between backups, or you are stuck backing up a single huge
file, and then it looks like there is no progress. Perhaps a better
indicator would be based on MB's processed compared to the size of the
previous backup. I'm sure patches are welcomed :)
[quote]The more I think about it, the more I want XferLOG
uncompressed/unbuffered, but also structured in a way so that it
starts a new line when it inspects a file, and then finishes the
line with details and the verdict (same, create, link, …)
[/quote]Feel free to write a patch to do what you want, but I expect patches to
BPC v3.x are unlikely to be added at this stage, unless they fix actual
problems (ie, preventing backups from working).

Remember in the majority of cases, you won't be watching backups, they
are something that *just happens*, and later you will come along and
verify they did happen, or restore some files. So "watching" a backup in
progress isn't a high priority...
[quote]
also sprach Tony Schreiner <anthony.schreiner < at > bc.edu> [2016-08-04 15]:
[quote]Also on the backup host, you can get the process id of the current dump
processes (there will be two per host during file transfer), and do

(sudo) ls -l /proc/{pid1,pid2}/fd

if a file is being written to backup it will show up in this list. But be
aware that there are times (sometimes long) when files are not being written
[/quote]What happens during those times?
[/quote]Backing up a single large (modified) file requires the server to
de-compress the original file, and then add the changes from the remote.
I'm not sure why, but BPC v3 seems to be rather in-efficient at this
process. This is one of the reasons I tend to split large remote files
on the remote side prior to BPC (eg, VM images, sql dumps, etc), (the
other reason is that most chunks will be the same, and so it saves disk
space on the BPC side, improves rsync bandwidth consumption, etc).

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-05 01]:
[quote]Maybe try this:
tail -f -n +0 blah.log | /usr/share/backuppc/bin/BackupPC_zcat -
You need to include the beginning of the file or else it won't
detect the compression header.... Also, the - frequently means use
stdin when a filename parameter is required, it may or may not be
needed.
[/quote]
This makes perfect sense and works; thanks!

[quote]Remember in the majority of cases, you won't be watching backups,
they are something that *just happens*, and later you will come
along and verify they did happen, or restore some files. So
"watching" a backup in progress isn't a high priority...
[/quote]
Well, yeah, except at the moment, I can't really rely on BackupPC to
"just do it". It's not properly tuned, and I am trying to drill down
towards the problem's root.

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

save the plankton - eat a whale.

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/