Welcome! » Log In » Create A New Profile

Choice of compression algorithm for logs

Posted by Anonymous 
Choice of compression algorithm for logs
August 04, 2016 12:36PM
Hello,

the fact that BackupPC compresses log files using zlib and requires
/usr/share/backuppc/bin/BackupPC_zcat for their uncompression is
a bit of a nuisance, not only when log files are being
sync'd/analysed on a system where there is no BackupPC installed.

I also can't find a suitable decompressor in a Debian package,
especially not one supporting reading from stdin.

Why aren't we just using standard gzip or bzip2 or xz, for which
decompressors exist on pretty much every Unix system?

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

if you see an onion ring -- answer it!

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Choice of compression algorithm for logs
August 04, 2016 04:10PM
On 05/08/16 05:33, martin f krafft wrote:
[quote]Hello,

the fact that BackupPC compresses log files using zlib and requires
/usr/share/backuppc/bin/BackupPC_zcat for their uncompression is
a bit of a nuisance, not only when log files are being
sync'd/analysed on a system where there is no BackupPC installed.

I also can't find a suitable decompressor in a Debian package,
especially not one supporting reading from stdin.

Why aren't we just using standard gzip or bzip2 or xz, for which
decompressors exist on pretty much every Unix system?
[/quote]I'm pretty sure there is a backuppc package for debian :)

If you really want a third party tool, and want that packaged for
debian, then I think you will need to be the one to do it/make it
happen. So far, everyone else has been happy without it (or happy enough
to not actually do it).

PS, we are using BackupPC's own compression tools because:

a) We know they will exist
b) We need to use them for the data, so might as well also use them for
the logs which will compress really well

You are free to make a patch that will use a different compression tool,
and allow that to be configured, perhaps others will appreciate it also.

Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Choice of compression algorithm for logs
August 05, 2016 12:19AM
also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-05 01]:
[quote][quote]Why aren't we just using standard gzip or bzip2 or xz, for which
decompressors exist on pretty much every Unix system?
[/quote]I'm pretty sure there is a backuppc package for debian :)
[/quote]
Oh yes, sure, but do I want/need to install it on all systems that
I might use to read logs?

[quote]PS, we are using BackupPC's own compression tools because:

a) We know they will exist
b) We need to use them for the data, so might as well also use them for
the logs which will compress really well
[/quote]
gzip will exist on a Unix system even after hell froze over… and
it's by far the more intuitive to use, with integration in editors,
viewers, mc, etc..

[quote]You are free to make a patch that will use a different compression
tool, and allow that to be configured, perhaps others will
appreciate it also.
[/quote]
I'll need to investigate this possibility. Generally, I try to avoid
Perl because I just don't get it. If you have any pointers, that'd
be great!

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

http://lavender.cime.net/~ricky/badgers.txt

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Choice of compression algorithm for logs
August 06, 2016 09:21AM
On Fri, Aug 5, 2016 at 2:17 AM, martin f krafft <madduck < at > madduck.net> wrote:
[quote]also sprach Adam Goryachev <mailinglists < at > websitemanagers.com.au> [2016-08-05 01]:
[quote][quote]Why aren't we just using standard gzip or bzip2 or xz, for which
decompressors exist on pretty much every Unix system?
[/quote]I'm pretty sure there is a backuppc package for debian :)
[/quote]
Oh yes, sure, but do I want/need to install it on all systems that
I might use to read logs?
[/quote]
Why is it likely that you would want to read backuppc logs on systems
that don't have backuppc installed. And why not do it through the web
interface where you don't need to care about how it is uncompressed?

[quote][quote]You are free to make a patch that will use a different compression
tool, and allow that to be configured, perhaps others will
appreciate it also.
[/quote]
I'll need to investigate this possibility. Generally, I try to avoid
Perl because I just don't get it. If you have any pointers, that'd
be great!
[/quote]
You can use a variety of different styles when writing perl. It can
look like C, shell, awk, or a few other languages so it is fairly easy
to write if you already know some computer language. The hard part
comes when you try to understand someone else's code where they used a
different style - or your own code from a few years ago...

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Choice of compression algorithm for logs
August 06, 2016 10:47PM
also sprach Les Mikesell <lesmikesell < at > gmail.com> [2016-08-06 18]:
[quote]Why is it likely that you would want to read backuppc logs on
systems that don't have backuppc installed.
[/quote]
Well, I collect all logs to a central location, but sure, this isn't
the normal usecase.

[quote]And why not do it through the web interface where you don't need
to care about how it is uncompressed?
[/quote]
Because the web interface is dreadfully slow and doesn't provide
grep, other Unix tools, nor shell automation.

[quote][quote]I'll need to investigate this possibility. Generally, I try to
avoid Perl because I just don't get it. If you have any
pointers, that'd be great!
[/quote]
You can use a variety of different styles when writing perl. It
can look like C, shell, awk, or a few other languages so it is
fairly easy to write if you already know some computer language.
The hard part comes when you try to understand someone else's code
where they used a different style - or your own code from a few
years ago...
[/quote]
Yeah, exactly… or when trying to figure out where to add code to an
existing project. ;)

--
< at > martinkrafft | http://madduck.net/ | http://two.sentenc.es/

"it is easier to be a lover than a husband for the simple reason
that it is more difficult to be witty every day
than to say pretty things from time to time."
-- honoré de balzac

spamtraps: madduck.bogus < at > madduck.net

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Choice of compression algorithm for logs
August 07, 2016 10:09AM
On Sun, Aug 7, 2016 at 12:45 AM, martin f krafft <madduck < at > madduck.net> wrote:
[quote]also sprach Les Mikesell <lesmikesell < at > gmail.com> [2016-08-06 18]:
[quote]Why is it likely that you would want to read backuppc logs on
systems that don't have backuppc installed.
[/quote]
Well, I collect all logs to a central location, but sure, this isn't
the normal usecase.
[/quote]
I guess if I were doing that, I'd likely use the backuppc host as the
place to collect stuff... Or NFS mount the collection.

[quote][quote]And why not do it through the web interface where you don't need
to care about how it is uncompressed?
[/quote]
Because the web interface is dreadfully slow and doesn't provide
grep, other Unix tools, nor shell automation.
[/quote]
I always had decent performance from the web interface but I think the
CentOS version I used ran under mod_perl. Sometimes the browser
search feature is handy. But, if you are into shell automation, why
not uncompress on the backuppc server where the tool is available
before moving them - or do it on the fly? Just some lazy sysadmin
ideas. It's not really any harder to run BackupPC_zcat remotely than
cat or cp and only slightly harder than using rsync or scp.

[quote][quote][quote]I'll need to investigate this possibility. Generally, I try to
avoid Perl because I just don't get it. If you have any
pointers, that'd be great!
[/quote]
You can use a variety of different styles when writing perl. It
can look like C, shell, awk, or a few other languages so it is
fairly easy to write if you already know some computer language.
The hard part comes when you try to understand someone else's code
where they used a different style - or your own code from a few
years ago...
[/quote]
Yeah, exactly… or when trying to figure out where to add code to an
existing project. ;)
[/quote]
Good perl programmers (not me...) usually encapsulate high level
operations into library modules, keeping the main program flow as a
compact set of calls to the modules. To make changes you have your
choice of adding options to the module code that does the work or
replacing the call completely with something different.

--
Les Mikesell
lesmikesell < at > gmail.com

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Choice of compression algorithm for logs
August 07, 2016 12:51PM
Hi there,

On Sun, 7 Aug 2016, Les Mikesell wrote:

[quote]You can use a variety of different styles when writing perl. It can
look like C, shell, awk, or ...
[/quote]
About fifteen years ago I was contracting on an early Web project for
a legal company in Los Angeles with about four million users. We used
Apache, mod_perl and a number of Perl toolkits, some rather obscure.

One day, one of the more technical managers glanced over my shoulder
at some Perl code as I was writing it. He said, "You know, when you
write Perl, I can understand it!"

I took that as the greatest of compliments.

I replied, "That's because I'm a C programmer." I've always regretted
not saying that I believe that it's very important that code should be
easy to understand. This is something that's been obvious to me ever
since I started programming the Motorola 6800 in assembler in the mid-
1970s, even if only because I had to persuade my boss that it would be
sane to let my new-fangled gadgets loose in a nuclear power plant.

I always strive for code to be easy to understand. When I go back to
it a decade or two later (it happens) I can still understand it.

--

73,
Ged.

------------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
BackupPC-users < at > lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Sorry, only registered users may post in this forum.

Click here to login