Deprecated: __autoload() is deprecated, use spl_autoload_register() instead in /home/pbuc/public_html/forum/mods/ext_phorummail/ezc/Base/src/ezc_bootstrap.php on line 36

Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; KeyCAPTCHA_CLASS has a deprecated constructor in /home/pbuc/public_html/forum/mods/keycaptcha/keycaptcha.php on line 108
rsnapshot and find
Welcome! » Log In » Create A New Profile

rsnapshot and find

Posted by Anonymous 
rsnapshot and find
October 10, 2014 06:47PM
Hi everyone,

Sorry for bringing this old topic out of graveyard, but I want to share my backup pet-project script:

[url=https://github.com/administratorius/LitBackup]https://github.com/administratorius/LitBackup[/url]

It has all the features we discussed in this thread.

On Mon, Jul 28, 2014 at 3:55 PM, David Cantrell <david < at > cantrell.org.uk ([email]david < at > cantrell.org.uk[/email])> wrote:
[quote]On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:

[quote]Backing up large amounts of files stresses SATA drives, so I thought - why
not use "find" on the server with SAS disks to find files and directories
modified during, say, last 24 hours and give the list to "rsync". That way
backup job takes much much less time - ran some tests. Do you see any cons
doing backups this way?
[/quote]
Yes. You won&#39;t be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget [url=http://alfriston2010.barnyard.co.uk/hall/index.html]http://alfriston2010.barnyard.co.uk/hall/index.html[/url]
...
$ ls -l index.html
-rw-r--r--    1 david    david        1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified several
days ago.

But in any case, I&#39;m not sure what "stress" you&#39;re trying to avoid.
You&#39;ll have to grovel over SAS disks just as much as SATA disks to find
what&#39;s changed.

[quote]Second question: is it possible to implement this using rsnapshot?
[/quote]
No.

--
David Cantrell | Hero of the Information Age

    Human Rights left unattended may be removed,
    destroyed, or damaged by the security services.

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
[url=http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

--
--

Vytenis

+370 674 47347
rsnapshot and find
October 10, 2014 06:52PM
Hi David,

Somehow I have missed Your post, sorry :)

[quote]On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:
[quote]Backing up large amounts of files stresses SATA drives, so I thought - why
not use "find" on the server with SAS disks to find files and directories
modified during, say, last 24 hours and give the list to "rsync". That way
backup job takes much much less time - ran some tests. Do you see any cons
doing backups this way?
[/quote]
Yes. You won&#39;t be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget [url=http://alfriston2010.barnyard.co.uk/hall/index.html]http://alfriston2010.barnyard.co.uk/hall/index.html[/url]
...
$ ls -l index.html
-rw-r--r--    1 david    david        1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified several
days ago.
[/quote]

I&#39;m checking "ctime", Your "ls" displays "mtime", which is old. Try using "stat index.html"
 
[quote]
But in any case, I&#39;m not sure what "stress" you&#39;re trying to avoid.
You&#39;ll have to grovel over SAS disks just as much as SATA disks to find
what&#39;s changed.

[/quote]

Yes, but ONLY SAS disks. Full rsync checks both SAS and SATA. Feeling weird trying to explain this for the 10-th time online and my coleagues :)

--
Vytenis
rsnapshot and find
October 10, 2014 08:34PM
On Fri, 10 Oct 2014 21:45:23 +0300
Vytenis A <vytenis.adm < at > gmail.com> wrote:

[quote]Hi everyone,

Sorry for bringing this old topic out of graveyard, but I want to
share my backup pet-project script:

https://github.com/administratorius/LitBackup

It has all the features we discussed in this thread.

On Mon, Jul 28, 2014 at 3:55 PM, David Cantrell <david < at > cantrell.org.uk>
wrote:

[quote]On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:

[quote]Backing up large amounts of files stresses SATA drives, so I
thought -
[/quote]why
[quote]not use "find" on the server with SAS disks to find files and
directories modified during, say, last 24 hours and give the list
to "rsync". That
[/quote]way
[quote]backup job takes much much less time - ran some tests. Do you see
any
[/quote]cons
[quote]doing backups this way?
[/quote]
Yes. You won't be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget http://alfriston2010.barnyard.co.uk/hall/index.html
...
$ ls -l index.html
-rw-r--r-- 1 david david 1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified
several days ago.

But in any case, I'm not sure what "stress" you're trying to avoid.
You'll have to grovel over SAS disks just as much as SATA disks to
find what's changed.

[quote]Second question: is it possible to implement this using rsnapshot?
[/quote]
No.

--
David Cantrell | Hero of the Information Age

Human Rights left unattended may be removed,
destroyed, or damaged by the security services.

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.

http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[/quote]

[/quote]
Glad to see you posting your efforts. I've looked over your code.

Pretty 'standard' sh code, even if not taking advantage of any actual
bash. You maintain an sh-only design (for compatibility to older
unices?), but specifically spell out /bin/bash in the shebang. Is there
a reason? Bash has a lot of features you are not using.

Q: should 'config/backup-excludes.cfg' have run/ and run/** in there as
well?

Some observations, designed to help and inform, not to be a prick:

cat /dev/null > $FILESLIST

can be expressed simply as:

[quote]$FILELIST
[/quote]

generally speaking, this construct should be avoided:

cat $FILESLIST.tmp|while read line; do :; done;

preferred format:

while read line; do :; done < FILELIST.tmp

reason being, is that your loop happens in a subshell, and any vars set
in the loop cannot be accessed *after* the loop terminates. I know
that's not actually needed in this case, but just sayin... it's just a
better habit to use this format.

also, this construct:

if [ "x$FILESLIST" == "x" ] || [ "x$LASTRUNMINS" == "x" ]
then
echo "wrong parameters given - please don't run this manually"|log
exit 1
fi

is simpler and cleaner as:

[[ ${FILELIST} && ${LASTRUNMINS} ]] || {
echo "wrong parameters given - please don't run this manually"|log
exit 1
}

no equalities to random letters or 'if...then...fi' required.

also, might want to pull out your logging functions into a common
sourced lib, rather than duplicating them.

sh is pretty limiting and generally pretty ugly - bash has a lot of
useful features you might want to examine.
readarray, [[, and ${BASH_REMATCH[ < at > ]} being just a few of them.

--
Regards,
Christopher Barry

Random geeky fortune:
Q: Why was Stonehenge abandoned?
A: It wasn't IBM compatible.

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 10, 2014 09:57PM
Hi Christopher,

I&#39;m not a developer and there is a lot of space for improvement.

[quote]

Glad to see you posting your efforts. I&#39;ve looked over your code.

Pretty &#39;standard&#39; sh code, even if not taking advantage of any actual
bash. You maintain an sh-only design (for compatibility to older
unices?), but specifically spell out /bin/bash in the shebang. Is there
a reason? Bash has a lot of features you are not using.

[/quote]

Regarding "bash" - no, there is no reason to put it there. Looks like I have to read some "bash" tutorials :) There are many ways to do one thing or another, it&#39;s difficult to choose the right one. Or even to find them.
 
[quote]
Q: should &#39;config/backup-excludes.cfg&#39; have run/ and run/** in there as
well?

[/quote]

Yes, that&#39;s a good idea.

 
[quote]
Some observations, designed to help and inform, not to be a prick:

 cat /dev/null > $FILESLIST

can be expressed simply as:

 > $FILELIST

[/quote]

Yes, I know that, but thought "cat /dev/null" would be more reliable.

 
[quote]
generally speaking, this construct should be avoided:

 cat $FILESLIST.tmp|while read line; do :; done;

preferred format:

 while read line; do :; done < FILELIST.tmp

reason being, is that your loop happens in a subshell, and any vars set
in the loop cannot be accessed *after* the loop terminates. I know
that&#39;s not actually needed in this case, but just sayin... it&#39;s just a
better habit to use this format.

[/quote]

That&#39;s a good insight. As I said - i&#39;m not fluent in bash scripting.

 
[quote]
also, this construct:

if [ "x$FILESLIST" == "x" ] || [ "x$LASTRUNMINS" == "x" ]
then
 echo "wrong parameters given - please don&#39;t run this manually"|log
 exit 1
fi

is simpler and cleaner as:

[[ ${FILELIST} && ${LASTRUNMINS} ]] || {
 echo "wrong parameters given - please don&#39;t run this manually"|log
 exit 1
}

no equalities to random letters or &#39;if...then...fi&#39; required.

[/quote]

Nice :)

 
[quote] also, might want to pull out your logging functions into a common
sourced lib, rather than duplicating them.

[/quote]

Yea, logging is a pain here, still not very satisfied as it is

 
[quote] sh is pretty limiting and generally pretty ugly - bash has a lot of
useful features you might want to examine.
readarray, [[, and ${BASH_REMATCH[ < at > ]} being just a few of them.

[/quote]

Thanks for Your time and guidance. Currently I am thinking about one improvement, and then I&#39;ll jump to more advanced techniques.

--
Vytenis
rsnapshot and find
October 11, 2014 04:11AM
On Fri, Oct 10, 2014 at 09:50:34PM +0300, Vytenis A (vytenis.adm < at > gmail.com) wrote:

[quote]On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:
[/quote]
[quote][quote][quote]Backing up large amounts of files stresses SATA drives, so I
thought - why not use "find" on the server with SAS disks to
find files and directories modified during, say, last 24 hours
and give the list to "rsync".
[/quote][/quote][/quote]
[quote][quote]Yes. You won't be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget http://alfriston2010.barnyard.co.uk/hall/index.html
...
$ ls -l index.html
-rw-r--r-- 1 david david 1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified several
days ago.

[/quote]
I'm checking "ctime", Your "ls" displays "mtime", which is old. Try using
"stat index.html"
[/quote]
Even using cime you would miss files in some circumstances.
Consider this example:

mv /nobackup/subdir /important

where directory "/nobackup" isn't being backed up but "/important" is.
After the mv, files under subdir have same (old) ctime as before and
would not be backed up by your scheme.

--
Tapani Tarvainen

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 11, 2014 05:04AM
On Sat, Oct 11, 2014 at 12:55:39AM +0300, Vytenis A (vytenis.adm < at > gmail.com) wrote:

[quote]I'm not a developer and there is a lot of space for improvement.
[/quote]
I took a quick look at your code, too.
In the interest of promoting good shell coding, a few comments:

[quote][quote]cat /dev/null > $FILESLIST

can be expressed simply as:

[quote]$FILELIST
[/quote]

[/quote]Yes, I know that, but thought "cat /dev/null" would be more reliable.
[/quote]
It isn't. It only wastes (admittedly only a minuscule amount)
cpu resources by forking an extra process (cat).

Also:

echo $ < at > |sh; STATUS=$?

could be done simply as

$ < at > ; STATUS=$?

No need to fork an extra subshell there either.

[quote][quote]also, might want to pull out your logging functions into a common
sourced lib, rather than duplicating them.

[/quote]Yea, logging is a pain here, still not very satisfied as it is
[/quote]
They're actually buggy, too. What happens if you send
a multiline message to logerror?
If you don't intend to use multiline messages, you
should not use while-do at all. Probably using stdin
to pass the message isn't optimal in any case,
it's again forking extra processes without need:
pass the message as a parameter instead.
In general, I prefer using standard syslog mechanism,
i.e., logger, in such scripts.

A bit more serious bug is in your lock file handling:
there's a race condition, two backups started
at almost same time could both end up running
(first tests lock, finds it isn't there, second
does likewise before first creates the lock).
Or one could be started before one has finished
(you remove the lock before final rm -f - what
happens if another backup starts during it?).

Lock files should always be tested and created
with a single atomic operation. With (ba)sh,
the usual way is like this:

set -o noclobber
if ! echo $$ > $LOCKFILE 2>/dev/null ; then
# lock taken, check if process is running &c
fi
set +o noclobber

and removed only at the very end - usually also even
when the script is interrupted, e.g., with

trap "rm $LOCK" 0

immediately after creating the lock.

--
Tapani Tarvainen

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 11, 2014 05:09AM
I'm still curious as to why you think using a SATA drive "stresses" it
more than a SAS drive.
We have several thousand drives, ranging from 500gb to 6tb, and an
equal mixture of SAS and SATA.
I can pull a hardware replacement sheet, but I can tell you that we
haven't seen a difference.

kw

On Fri, Oct 10, 2014 at 10:45 AM, Vytenis A <vytenis.adm < at > gmail.com> wrote:
[quote]Hi everyone,

Sorry for bringing this old topic out of graveyard, but I want to share my
backup pet-project script:

https://github.com/administratorius/LitBackup

It has all the features we discussed in this thread.

On Mon, Jul 28, 2014 at 3:55 PM, David Cantrell <david < at > cantrell.org.uk>
wrote:
[quote]
On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:

[quote]Backing up large amounts of files stresses SATA drives, so I thought -
why
not use "find" on the server with SAS disks to find files and
directories
modified during, say, last 24 hours and give the list to "rsync". That
way
backup job takes much much less time - ran some tests. Do you see any
cons
doing backups this way?
[/quote]
Yes. You won't be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget http://alfriston2010.barnyard.co.uk/hall/index.html
...
$ ls -l index.html
-rw-r--r-- 1 david david 1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified several
days ago.

But in any case, I'm not sure what "stress" you're trying to avoid.
You'll have to grovel over SAS disks just as much as SATA disks to find
what's changed.

[quote]Second question: is it possible to implement this using rsnapshot?
[/quote]
No.

--
David Cantrell | Hero of the Information Age

Human Rights left unattended may be removed,
destroyed, or damaged by the security services.

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.

http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
[/quote]

--
--
Vytenis
+370 674 47347

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[/quote]
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 11, 2014 06:40AM
[quote] Even using cime you would miss files in some circumstances.
Consider this example:

mv /nobackup/subdir /important

where directory "/nobackup" isn&#39;t being backed up but "/important" is.
After the mv, files under subdir have same (old) ctime as before and
would not be backed up by your scheme.
[/quote]

This problem was discussed earlier in this thread. I have made a workaround to recursively rsync entire directory if it&#39;s ctime has changed, which basically downgrades performance. In Your example it will sync /important recursively. I have also [url=http://stackoverflow.com/questions/26305661/rsync-recursively-only-new-directories]asked on stackoverflow[/url] about this issue (how to recursively rsync only new directories)

 

--

Vytenis
rsnapshot and find
October 11, 2014 06:43AM
Thank You Tapaini for insights. I will look into perfecting this script. This list showed me that my coding skills aren&#39;t as good as I&#39;d like :)

On Sat, Oct 11, 2014 at 8:02 AM, Tapani Tarvainen <rsnapshot < at > tapanitarvainen.fi ([email]rsnapshot < at > tapanitarvainen.fi[/email])> wrote:
[quote]On Sat, Oct 11, 2014 at 12:55:39AM +0300, Vytenis A (vytenis.adm < at > gmail.com ([email]vytenis.adm < at > gmail.com[/email])) wrote:

[quote]I&#39;m not a developer and there is a lot of space for improvement.
[/quote]
I took a quick look at your code, too.
In the interest of promoting good shell coding, a few comments:

[quote][quote]  cat /dev/null > $FILESLIST

can be expressed simply as:

  > $FILELIST

[/quote]Yes, I know that, but thought "cat /dev/null" would be more reliable.
[/quote]
It isn&#39;t. It only wastes (admittedly only a minuscule amount)
cpu resources by forking an extra process (cat).

Also:

echo $ < at > |sh; STATUS=$?

could be done simply as

$ < at > ; STATUS=$?

No need to fork an extra subshell there either.

[quote][quote]also, might want to pull out your logging functions into a common
sourced lib, rather than duplicating them.

[/quote]Yea, logging is a pain here, still not very satisfied as it is
[/quote]
They&#39;re actually buggy, too. What happens if you send
a multiline message to logerror?
If you don&#39;t intend to use multiline messages, you
should not use while-do at all. Probably using stdin
to pass the message isn&#39;t optimal in any case,
it&#39;s again forking extra processes without need:
pass the message as a parameter instead.
In general, I prefer using standard syslog mechanism,
i.e., logger, in such scripts.

A bit more serious bug is in your lock file handling:
there&#39;s a race condition, two backups started
at almost same time could both end up running
(first tests lock, finds it isn&#39;t there, second
does likewise before first creates the lock).
Or one could be started before one has finished
(you remove the lock before final rm -f - what
happens if another backup starts during it?).

Lock files should always be tested and created
with a single atomic operation. With (ba)sh,
the usual way is like this:

set -o noclobber
if ! echo $$ > $LOCKFILE 2>/dev/null ; then
  # lock taken, check if process is running &c
fi
set +o noclobber

and removed only at the very end - usually also even
when the script is interrupted, e.g., with

trap "rm $LOCK" 0

immediately after creating the lock.

--
Tapani Tarvainen

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
[url=http://p.sf.net/sfu/Zoho]http://p.sf.net/sfu/Zoho[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

--
--

Vytenis

+370 674 47347
rsnapshot and find
October 11, 2014 07:06AM
Hi Ken,

Since this is a question everybody asks I decided to update my "README.txt". If anyone who understands this knows how to explain it simplier - it would be nice to hear.

 Here&#39;s the snippet from "README.txt":

<...>
Everyone asks: why this is faster than regular "rsync"?
When rsync syncs two directories "/source" and "/destination" on hosts "srchost" and "dsthost", it compares "ctime" and size on both hosts. It means that on "dsthost" each file generates an "fstat" syscall, which accesses disks.
My approach is to "find" all files on "srchost" and do only WRITE operations to "dsthost" saving tons of read operations on "dsthost".
Example: imagine that on "srchost" only "/var/log/messages" has changed. If I want to backup "/" I will basically compare whole "srchost" with "dsthost" server for changes and copy only "/var/log/messages". In my case "dsthost" would get a list of files to backup and do just one I/O operation - rsync "/var/log/messages"<...>

On Sat, Oct 11, 2014 at 8:06 AM, Ken Woods <kenwoods < at > gmail.com ([email]kenwoods < at > gmail.com[/email])> wrote:
[quote]I&#39;m still curious as to why you think using a SATA drive "stresses" it
more than a SAS drive.
We have several thousand drives, ranging from 500gb to 6tb, and an
equal mixture of SAS and SATA.
I can pull a hardware replacement sheet, but I can tell you that we
haven&#39;t seen a difference.

kw

On Fri, Oct 10, 2014 at 10:45 AM, Vytenis A <vytenis.adm < at > gmail.com ([email]vytenis.adm < at > gmail.com[/email])> wrote:
[quote]Hi everyone,

Sorry for bringing this old topic out of graveyard, but I want to share my
backup pet-project script:

[url=https://github.com/administratorius/LitBackup]https://github.com/administratorius/LitBackup[/url]

It has all the features we discussed in this thread.

On Mon, Jul 28, 2014 at 3:55 PM, David Cantrell <david < at > cantrell.org.uk ([email]david < at > cantrell.org.uk[/email])>
wrote:
[quote]
On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:

[quote]Backing up large amounts of files stresses SATA drives, so I thought -
why
not use "find" on the server with SAS disks to find files and
directories
modified during, say, last 24 hours and give the list to "rsync". That
way
backup job takes much much less time - ran some tests. Do you see any
cons
doing backups this way?
[/quote]
Yes. You won&#39;t be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget [url=http://alfriston2010.barnyard.co.uk/hall/index.html]http://alfriston2010.barnyard.co.uk/hall/index.html[/url]
...
$ ls -l index.html
-rw-r--r--    1 david    david        1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified several
days ago.

But in any case, I&#39;m not sure what "stress" you&#39;re trying to avoid.
You&#39;ll have to grovel over SAS disks just as much as SATA disks to find
what&#39;s changed.

[quote]Second question: is it possible to implement this using rsnapshot?
[/quote]
No.

--
David Cantrell | Hero of the Information Age

     Human Rights left unattended may be removed,
     destroyed, or damaged by the security services.

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.

[url=http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]
[/quote]

--
--
Vytenis
[url=tel]+370 674 47347[/url]

[/quote]

[quote]------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
[url=http://p.sf.net/sfu/Zoho]http://p.sf.net/sfu/Zoho[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

[/quote]

--
--

Vytenis

+370 674 47347
rsnapshot and find
October 11, 2014 08:15AM
On Sat, Oct 11, 2014 at 09:38:24AM +0300, Vytenis A (vytenis.adm < at > gmail.com) wrote:

[quote][quote]mv /nobackup/subdir /important

where directory "/nobackup" isn't being backed up but "/important" is.
After the mv, files under subdir have same (old) ctime as before and
would not be backed up by your scheme.

[/quote]
This problem was discussed earlier in this thread. I have made a workaround
to recursively rsync entire directory if it's ctime has changed, which
basically downgrades performance.
[/quote]
Hmm. Yes, that should work. And it should only take adding options to find
to stop descending into directories whose ctime has changed. Clever.

But it would indeed slow things down, possibly significantly:
directory ctime can change for many reasons, and you'd rsync
them entirely even when it wouldn't be necessary.
How big the effect would be, hard to tell without testing.

[quote]I have also asked on stackoverflow
http://stackoverflow.com/questions/26305661/rsync-recursively-only-new-directories
about this issue (how to recursively rsync only new directories)
[/quote]
I don't think rsync can do that.

One thing you could do (and what many "big" backup programs do)
is to keep the recursive directory listing from previous run and
back up everything that's in a new location (in addition to
those with changed ctime). But it would complicate your program
quite a lot.

--
Tapani Tarvainen

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 11, 2014 08:51AM
Hallo, Vytenis,

Du meintest am 11.10.14:

[quote]Thank You Tapaini for insights. I will look into perfecting this
script. This list showed me that my coding skills aren't as good as
I'd like :)
[/quote]
[quote]On Sat, Oct 11, 2014 at 8:02 AM, Tapani Tarvainen <
rsnapshot < at > tapanitarvainen.fi> wrote:
[/quote]
[quote][quote]On Sat, Oct 11, 2014 at 12:55:39AM +0300, Vytenis A
(vytenis.adm < at > gmail.com) wrote:

[quote]I'm not a developer and there is a lot of space for improvement.
[/quote]
I took a quick look at your code, too.
In the interest of promoting good shell coding, a few comments:

[quote][quote]cat /dev/null > $FILESLIST

can be expressed simply as:

[quote]$FILELIST
[/quote]

[/quote]Yes, I know that, but thought "cat /dev/null" would be more
reliable.
[/quote]
It isn't. It only wastes (admittedly only a minuscule amount)
cpu resources by forking an extra process (cat).

Also:

echo $ < at > |sh; STATUS=$?

could be done simply as

$ < at > ; STATUS=$?

No need to fork an extra subshell there either.

[quote][quote]also, might want to pull out your logging functions into a common
sourced lib, rather than duplicating them.

[/quote]Yea, logging is a pain here, still not very satisfied as it is
[/quote]
They're actually buggy, too. What happens if you send
a multiline message to logerror?
If you don't intend to use multiline messages, you
should not use while-do at all. Probably using stdin
to pass the message isn't optimal in any case,
it's again forking extra processes without need:
pass the message as a parameter instead.
In general, I prefer using standard syslog mechanism,
i.e., logger, in such scripts.

A bit more serious bug is in your lock file handling:
there's a race condition, two backups started
at almost same time could both end up running
(first tests lock, finds it isn't there, second
does likewise before first creates the lock).
Or one could be started before one has finished
(you remove the lock before final rm -f - what
happens if another backup starts during it?).

Lock files should always be tested and created
with a single atomic operation. With (ba)sh,
the usual way is like this:

set -o noclobber
if ! echo $$ > $LOCKFILE 2>/dev/null ; then
# lock taken, check if process is running &c
fi
set +o noclobber

and removed only at the very end - usually also even
when the script is interrupted, e.g., with

trap "rm $LOCK" 0

immediately after creating the lock.

--
Tapani Tarvainen

--------------------------------------------------------------------
---------- Meet PCI DSS 3.0 Compliance Requirements with EventLog
Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box
PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance?
Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5
with EventLog Analyzer http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[/quote][/quote]

[quote]--
--
Vytenis
+370 674 47347
[/quote]
Can you please stop

- TOFU
- HTML

Thank you!

Viele Gruesse!
Helmut

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 11, 2014 02:32PM
I'm still missing the part were SAS begets stressed less than SATA.

On Oct 10, 2014, at 23:03, Vytenis A <vytenis.adm < at > gmail.com ([email]vytenis.adm < at > gmail.com[/email])> wrote:

[quote]Hi Ken,

Since this is a question everybody asks I decided to update my "README.txt". If anyone who understands this knows how to explain it simplier - it would be nice to hear.

Here's the snippet from "README.txt":

<...>
Everyone asks: why this is faster than regular "rsync"?
When rsync syncs two directories "/source" and "/destination" on hosts "srchost" and "dsthost", it compares "ctime" and size on both hosts. It means that on "dsthost" each file generates an "fstat" syscall, which accesses disks.
My approach is to "find" all files on "srchost" and do only WRITE operations to "dsthost" saving tons of read operations on "dsthost".
Example: imagine that on "srchost" only "/var/log/messages" has changed. If I want to backup "/" I will basically compare whole "srchost" with "dsthost" server for changes and copy only "/var/log/messages". In my case "dsthost" would get a list of files to backup and do just one I/O operation - rsync "/var/log/messages"<...>

On Sat, Oct 11, 2014 at 8:06 AM, Ken Woods <kenwoods < at > gmail.com ([email]kenwoods < at > gmail.com[/email])> wrote:
[quote]I'm still curious as to why you think using a SATA drive "stresses" it
more than a SAS drive.
We have several thousand drives, ranging from 500gb to 6tb, and an
equal mixture of SAS and SATA.
I can pull a hardware replacement sheet, but I can tell you that we
haven't seen a difference.

kw

On Fri, Oct 10, 2014 at 10:45 AM, Vytenis A <vytenis.adm < at > gmail.com ([email]vytenis.adm < at > gmail.com[/email])> wrote:
[quote]Hi everyone,

Sorry for bringing this old topic out of graveyard, but I want to share my
backup pet-project script:

[url=https://github.com/administratorius/LitBackup]https://github.com/administratorius/LitBackup[/url]

It has all the features we discussed in this thread.

On Mon, Jul 28, 2014 at 3:55 PM, David Cantrell <david < at > cantrell.org.uk ([email]david < at > cantrell.org.uk[/email])>
wrote:
[quote]
On Fri, Jul 25, 2014 at 11:40:37PM +0300, Vytenis A wrote:

[quote]Backing up large amounts of files stresses SATA drives, so I thought -
why
not use "find" on the server with SAS disks to find files and
directories
modified during, say, last 24 hours and give the list to "rsync". That
way
backup job takes much much less time - ran some tests. Do you see any
cons
doing backups this way?
[/quote]
Yes. You won't be backing up all the changed files. Consider this:

$ date
Mon Jul 28 13:50:37 BST 2014
$ wget [url=http://alfriston2010.barnyard.co.uk/hall/index.html]http://alfriston2010.barnyard.co.uk/hall/index.html[/url]
...
$ ls -l index.html
-rw-r--r-- 1 david david 1763 Jul 23 23:01 index.html

Note that the brand new file appears to have been last modified several
days ago.

But in any case, I'm not sure what "stress" you're trying to avoid.
You'll have to grovel over SAS disks just as much as SATA disks to find
what's changed.

[quote]Second question: is it possible to implement this using rsnapshot?
[/quote]
No.

--
David Cantrell | Hero of the Information Age

Human Rights left unattended may be removed,
destroyed, or damaged by the security services.

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.

[url=http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk]http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]
[/quote]

--
--
Vytenis
[url=tel]+370 674 47347[/url]

[/quote]

[quote]------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
[url=http://p.sf.net/sfu/Zoho]http://p.sf.net/sfu/Zoho[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

[/quote]

--
--

Vytenis

+370 674 47347

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
[url=http://p.sf.net/sfu/Zoho]http://p.sf.net/sfu/Zoho[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]
rsnapshot and find
October 11, 2014 04:49PM
On Sat, 11 Oct 2014 06:29:27 -0800
Ken Woods <kenwoods < at > gmail.com> wrote:

[quote]I'm still missing the part were SAS begets stressed less than SATA.
[/quote]
please use standard mailing list bottom posting

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
rsnapshot and find
October 11, 2014 04:57PM
On Sat, 11 Oct 2014 06:29:27 -0800
Ken Woods <kenwoods < at > gmail.com> wrote:

[quote]I'm still missing the part were SAS begets stressed less than SATA.

[/quote]
My guess, having built many backup systems in my career, is the
nearline backup server has a ton of cheaper SATA disks in it, and backs
up many production servers with SAS disks.

These SATA disks are typically 5400rpm to 7200rpm. The servers have
higher-end SAS disks running at 10k-15k rpm.

By running the find operation on the servers, and then telling the
backup server to simply pull these specific files in the list, a
non-trivial amount of disk IO is eliminated on the backup server. This
both speeds up the total operation, and does not bang on the cheaper
disks as hard.

--
Regards,
Christopher Barry

Random geeky fortune:
"It is the creationists who blasphemously are claiming that God is
cheating us in a stupid way."
-- J. W. Nienhuys

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss
Sorry, only registered users may post in this forum.

Click here to login