Weekly backup bombing out not finding daily.6

I have rsnapshot set up to do 7 daily backups and a weekly backup. The target for the backups are on a remote NAS which gets mounted/unmounted by a script run as cmd_preexec and cmd_postexec.

This is all working fine for the daily backups. But when it comes to do the weekly backup I get this error:

   /mnt/pme-nas2/Backups/daily.6 not present (yet), nothing to copy

If I mount the NAS manually before running rsnapshot the weekly backup works OK. 

As the remote NAS gets mounted properly for the daily backups, I'm wondering why it doesn't for the weekly. Does the weekly copy section happen before the cmd_preexec is run, perhaps? This is how it's looking to me because I have echo statements in the cmd_preexec that don't show any output before I ge the "not present yet" error. Any suggestions for how I can resolve this?

[quote]I have made similar mistake in the past:  I started the weekly before the daily has completed.

You want to make sure the daily job is completed first -- In my case:

45 23 * * 0-6 /usr/bin/rsnapshot daily # daily backup ran at 11:45pm
10 2 * * 7 /usr/bin/rsnapshot weekly # Weekly backup at 3:10am Sunday

[/quote]

Thanks Anthony, that's interesting. But I'm not yet convinced it's the root cause of my problem. I set the weekly up to run BEFORE the daily based on some very clear advice I read on a QNAP forum. I also read it on the rsnapshot HOW itself at [url=http://rsnapshot.org/rsnapshot/docs/docbook/rest.html]http://rsnapshot.org/rsnapshot/docs/docbook/rest.html[/url]. It says "It is usually a good idea to schedule the larger intervals to run a bit before the lower ones. For example, in the crontab above, notice that daily runs 30 minutes before hourly. This helps prevent race conditions where the daily would try to run before the hourly job had finished. This same strategy should be extended so that a weekly entry would run before the daily and so on".

So now I'm confused. Is my problem really because I'm running weekly before daily? I don't actually see how it can be because the daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I have scheduled it to run BEFORE daily. In what order do other folks reading this set up daily vs weekly?

I am very suspicious that the real problem is that when running any interval which isn't the most frequent, the rsnapshot script doesn't run cmd_preexec (which in my case means the target for the backup isn't mounted).

 Can anyone confirm?

[quote]So now I'm confused. Is my problem really because I'm running weekly before daily? I don't actually see how it can be because the daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I have scheduled it to run BEFORE daily. In what order do other folks reading this set up daily vs weekly?

I am very suspicious that the real problem is that when running any interval which isn't the most frequent, the rsnapshot script doesn't run cmd_preexec (which in my case means the target for the backup isn't mounted).

 Can anyone confirm?

[/quote]

I just saw this which confirms my theory. For anyone in future...DO NOT use cmd_preexec to mount the target for the backups. While it seems like a logical place to do it, cmd_preexec only runs for the most frequent interval backups. So if, for example, you had daily, weekly and monthly, cmd_preexec will only be run for the daily backups.

On 14 May 2016 at 14:50, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote][quote]So now I&#39;m confused. Is my problem really because I&#39;m running weekly before daily? I don&#39;t actually see how it can be because the daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I have scheduled it to run BEFORE daily. In what order do other folks reading this set up daily vs weekly?

I am very suspicious that the real problem is that when running any interval which isn&#39;t the most frequent, the rsnapshot script doesn&#39;t run cmd_preexec (which in my case means the target for the backup isn&#39;t mounted).

 Can anyone confirm?

[/quote]

I just saw this which confirms my theory. For anyone in future...DO NOT use cmd_preexec to mount the target for the backups. While it seems like a logical place to do it, cmd_preexec only runs for the most frequent interval backups. So if, for example, you had daily, weekly and monthly, cmd_preexec will only be run for the daily backups.

[/quote]

Sorry forgot to paste the link I saw - [url=http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/]http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/[/url]

On Sat, May 14, 2016 at 12:52 AM, Phill Edwards <philledwards < at > gmail.com> wrote:
[quote]

On 14 May 2016 at 14:50, Phill Edwards <philledwards < at > gmail.com> wrote:
[quote][quote]
So now I'm confused. Is my problem really because I'm running weekly
before daily? I don't actually see how it can be because the
daily.0/1/2/3/4/5/6 are always there so the weekly should just work - so I
have scheduled it to run BEFORE daily. In what order do other folks reading
this set up daily vs weekly?

I am very suspicious that the real problem is that when running any
interval which isn't the most frequent, the rsnapshot script doesn't run
cmd_preexec (which in my case means the target for the backup isn't
mounted).

Can anyone confirm?
[/quote]

I just saw this which confirms my theory. For anyone in future...DO NOT
use cmd_preexec to mount the target for the backups. While it seems like a
logical place to do it, cmd_preexec only runs for the most frequent interval
backups. So if, for example, you had daily, weekly and monthly, cmd_preexec
will only be run for the daily backups.
[/quote]

Sorry forgot to paste the link I saw -
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/
[/quote]
Ouch. Also see my advice at the bottom of that thread. If I have to
play games like remounting an attached drive or NFS mount in "rw" mode
to run rsnapshot, I've tended to do it in a wrapper that can execute
"trap" functions if something fails horribly.

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[quote][quote][quote]I just saw this which confirms my theory. For anyone in future...DO NOT use cmd_preexec to mount the target for the backups. While it seems like a logical place to do it, cmd_preexec only runs for the most frequent interval backups. So if, for example, you had daily, weekly and monthly, cmd_preexec will only be run for the daily backups.

[/quote]

[/quote]

Sorry forgot to paste the link I saw - [url=http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/]http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/[/url]

[/quote]

For the benefit of the archive, I thought I would post the rsnapshot wrapper shell script that I wrote in case anyone finds it useful in future. And if anyone has any feedback on how it could be improved, please shout out! The script essentially mounts the remote backup target (assumes it&#39;s a NFS share) if it&#39;s not already mounted, then runs rsnapshot if that all works OK, and finally unmounts it. If you wanted to mount a non-NFS share it would be a simple change to the script.

#!/bin/sh
# -----------------------------------------------------------------------------
#
# Script: pme_rsnapshot_wrapper.sh
# Created by PME to act as a wrapper script for rsnapshot. It&#39;s called from
# cron and mounts the target NFS backup share, checks it&#39;s mounted and then
# runs rsnapshot. When rsnapshot&#39;s finished it unmounts the target share.
#
# This is instead of using cmd_prexec and cmd_postexec configs in rsnapshot
# because they&#39;re only run fior the most frequent interval backup (ie daily)
# so don&#39;t get run for weekly and monthly backups.
#
# Parameters:
#     $1 = rsnapshot backup interval: daily, weekly or monthly
#     $2 = "DUMMYRUN" if don&#39;t actually want the rsnapshot backup to run.
#
# Usage:
#     pme_rsnapshot_wrapper.sh daily|weekly|monthly [DUMMYRUN]
#
# Example command to mount NFS share on NAS2 is:
#     /bin/mount -t nfs remote.server.name:/mnt/sda2 /mnt/nas2/
#
# -----------------------------------------------------------------------------

# -----------------------------------------------------------------------------
# Set config variables
# -----------------------------------------------------------------------------

# Location for target of backups.
BKUP_TARGET="remote.server.name:/mnt/sda2"

# Mount point - don&#39;t put a / on end cos it will fail grep of /proc/mounts
MNT_POINT="/mnt/nas2"

# rsnapshot executable and path (this is on a QNAP NAS)
RSNAP_EXE="/share/MD0_DATA/.qpkg/Entware-ng/bin/rsnapshot"

# -----------------------------------------------------------------------------
# END OF CONFIG. Shouldn&#39;t need to change anything below here (unless you want
# to add a new backup interval).
# -----------------------------------------------------------------------------

# -----------------------------------------------------------------------------
# Function to mount remote backup target on local dir.
# -----------------------------------------------------------------------------
mountBackupTarget () {
   if grep -qs "$MNT_POINT" /proc/mounts; then
      echo "`date` pme_rsnapshot_wrapper.sh: INF: $MNT_POINT is already mounted"
      return 0
   else
      echo "`date` pme_rsnapshot_wrapper.sh: INF: mounting $BKUP_TARGET on $MNT_POINT"
      /bin/mount -t nfs $BKUP_TARGET $MNT_POINT
      return $?
   fi
}

# -----------------------------------------------------------------------------
# Function to UNmount backup target after backup finished.
# -----------------------------------------------------------------------------
unmountBackupTarget () {
   echo "`date` pme_rsnapshot_wrapper.sh: INF: unmounting $BKUP_TARGET"
   /bin/umount $BKUP_TARGET
}

# -----------------------------------------------------------------------------
# MAIN
# -----------------------------------------------------------------------------
# Check num of parameters passed is correct (should be at least 1)
if [ $# -lt 1 ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect number of params: $1=$1"
   echo "                             usage: pme_rsnapshot_wrapper.sh daily|weekly|monthly"
   exit 1

# Execute mount command
elif [ "$1" == "daily" ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot "$1" requested"

# Execute UNmount command
elif [ "$1" == "weekly" ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot "$1" requested"

# Execute UNmount command
elif [ "$1" == "monthly" ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot "$1" requested"

# Command line parameter was some other rubbish.
else
   echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect param "$1""
   echo "                             usage: pme_rsnapshot_wrapper.sh daily|weekly|monthly"
   exit 1
fi

# -----------------------------------------------------------------------------
# Mount remote backup target and run rsnapshot if successful.
# -----------------------------------------------------------------------------
mountBackupTarget
RETVAL=$?               # Gets return value of the mount command.

if [ "$RETVAL" -eq 0 ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: $BKUP_TARGET mounted OK"

   if [ "$2" == "DUMMYRUN" ]; then
      echo "`date` pme_rsnapshot_wrapper.sh: INF: $2 so rsnapshot not executed"
   else
      echo "`date` pme_rsnapshot_wrapper.sh: INF: running $RSNAP_EXE $1"
      $RSNAP_EXE $1     # Run rsnapshot backup
   fi

   unmountBackupTarget  # UNmount backup directory when rsnapshot completed.

else
   echo "`date` pme_rsnapshot_wrapper.sh: ERR: $BKUP_TARGET mount failed with code $RETVAL"
   exit 1
fi

I haven&#39;t really been following this thread but I have one question: are you running rsnapshot on the NAS or on the client(s)? The fact that you mount the NAS via NFS leads me to suspect the former. It&#39;s worth noting that this is by far the worst (i.e. least efficient) way to do it. I used to do it this way a few years ago because it seemed so natural, but in fact rsnapshot works much better if you run it on the NAS and connect to the client(s) via the rsync protocol. This is not very clear from the documentation - I guess the author just assumed it was obvious - but when I changed methods I saw a speedup by an order of magnitude.

The reason is that treating the NAS as if it were local means that the receiving rsync (which is still running on the client machine!) is reading huge quantities of data in order to check for modified files. Doing it the other way means the receiver runs on the NAS and the network is only used to send summaries and changed files, with the resulting boost in efficiency. It also means you don&#39;t need NFS for backups (though it&#39;s still useful for restores).

Sorry of I&#39;ve misunderstood what you are doing, but I feel this point is worth emphasizing and should be included in the docs.

poc

On 19 May 2016 at 01:42, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote][quote][quote][quote]I just saw this which confirms my theory. For anyone in future...DO NOT use cmd_preexec to mount the target for the backups. While it seems like a logical place to do it, cmd_preexec only runs for the most frequent interval backups. So if, for example, you had daily, weekly and monthly, cmd_preexec will only be run for the daily backups.

[/quote]

[/quote]

Sorry forgot to paste the link I saw - [url=http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/]http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rsnapshot-24/mount-backup-directory-122004/[/url]

[/quote]

For the benefit of the archive, I thought I would post the rsnapshot wrapper shell script that I wrote in case anyone finds it useful in future. And if anyone has any feedback on how it could be improved, please shout out! The script essentially mounts the remote backup target (assumes it&#39;s a NFS share) if it&#39;s not already mounted, then runs rsnapshot if that all works OK, and finally unmounts it. If you wanted to mount a non-NFS share it would be a simple change to the script.

#!/bin/sh
# -----------------------------------------------------------------------------
#
# Script: pme_rsnapshot_wrapper.sh
# Created by PME to act as a wrapper script for rsnapshot. It&#39;s called from
# cron and mounts the target NFS backup share, checks it&#39;s mounted and then
# runs rsnapshot. When rsnapshot&#39;s finished it unmounts the target share.
#
# This is instead of using cmd_prexec and cmd_postexec configs in rsnapshot
# because they&#39;re only run fior the most frequent interval backup (ie daily)
# so don&#39;t get run for weekly and monthly backups.
#
# Parameters:
#     $1 = rsnapshot backup interval: daily, weekly or monthly
#     $2 = "DUMMYRUN" if don&#39;t actually want the rsnapshot backup to run.
#
# Usage:
#     pme_rsnapshot_wrapper.sh daily|weekly|monthly [DUMMYRUN]
#
# Example command to mount NFS share on NAS2 is:
#     /bin/mount -t nfs remote.server.name:/mnt/sda2 /mnt/nas2/
#
# -----------------------------------------------------------------------------

# -----------------------------------------------------------------------------
# Set config variables
# -----------------------------------------------------------------------------

# Location for target of backups.
BKUP_TARGET="remote.server.name:/mnt/sda2"

# Mount point - don&#39;t put a / on end cos it will fail grep of /proc/mounts
MNT_POINT="/mnt/nas2"

# rsnapshot executable and path (this is on a QNAP NAS)
RSNAP_EXE="/share/MD0_DATA/.qpkg/Entware-ng/bin/rsnapshot"

# -----------------------------------------------------------------------------
# END OF CONFIG. Shouldn&#39;t need to change anything below here (unless you want
# to add a new backup interval).
# -----------------------------------------------------------------------------

# -----------------------------------------------------------------------------
# Function to mount remote backup target on local dir.
# -----------------------------------------------------------------------------
mountBackupTarget () {
   if grep -qs "$MNT_POINT" /proc/mounts; then
      echo "`date` pme_rsnapshot_wrapper.sh: INF: $MNT_POINT is already mounted"
      return 0
   else
      echo "`date` pme_rsnapshot_wrapper.sh: INF: mounting $BKUP_TARGET on $MNT_POINT"
      /bin/mount -t nfs $BKUP_TARGET $MNT_POINT
      return $?
   fi
}

# -----------------------------------------------------------------------------
# Function to UNmount backup target after backup finished.
# -----------------------------------------------------------------------------
unmountBackupTarget () {
   echo "`date` pme_rsnapshot_wrapper.sh: INF: unmounting $BKUP_TARGET"
   /bin/umount $BKUP_TARGET
}

# -----------------------------------------------------------------------------
# MAIN
# -----------------------------------------------------------------------------
# Check num of parameters passed is correct (should be at least 1)
if [ $# -lt 1 ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect number of params: $1=$1"
   echo "                             usage: pme_rsnapshot_wrapper.sh daily|weekly|monthly"
   exit 1

# Execute mount command
elif [ "$1" == "daily" ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot "$1" requested"

# Execute UNmount command
elif [ "$1" == "weekly" ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot "$1" requested"

# Execute UNmount command
elif [ "$1" == "monthly" ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: rsnapshot "$1" requested"

# Command line parameter was some other rubbish.
else
   echo "`date` pme_rsnapshot_wrapper.sh: ERR: Incorrect param "$1""
   echo "                             usage: pme_rsnapshot_wrapper.sh daily|weekly|monthly"
   exit 1
fi

# -----------------------------------------------------------------------------
# Mount remote backup target and run rsnapshot if successful.
# -----------------------------------------------------------------------------
mountBackupTarget
RETVAL=$?               # Gets return value of the mount command.

if [ "$RETVAL" -eq 0 ]; then
   echo "`date` pme_rsnapshot_wrapper.sh: INF: $BKUP_TARGET mounted OK"

   if [ "$2" == "DUMMYRUN" ]; then
      echo "`date` pme_rsnapshot_wrapper.sh: INF: $2 so rsnapshot not executed"
   else
      echo "`date` pme_rsnapshot_wrapper.sh: INF: running $RSNAP_EXE $1"
      $RSNAP_EXE $1     # Run rsnapshot backup
   fi

   unmountBackupTarget  # UNmount backup directory when rsnapshot completed.

else
   echo "`date` pme_rsnapshot_wrapper.sh: ERR: $BKUP_TARGET mount failed with code $RETVAL"
   exit 1
fi

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
[url=https://ad.doubleclick.net/ddm/clk/304595813;131938128;j]https://ad.doubleclick.net/ddm/clk/304595813;131938128;j[/url]
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net ([email]rsnapshot-discuss < at > lists.sourceforge.net[/email])
[url=https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss]https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss[/url]

[/quote]

[quote]I haven&#39;t really been following this thread but I have one question: are you running rsnapshot on the NAS or on the client(s)? The fact that you mount the NAS via NFS leads me to suspect the former. It&#39;s worth noting that this is by far the worst (i.e. least efficient) way to do it. I used to do it this way a few years ago because it seemed so natural, but in fact rsnapshot works much better if you run it on the NAS and connect to the client(s) via the rsync protocol. This is not very clear from the documentation - I guess the author just assumed it was obvious - but when I changed methods I saw a speedup by an order of magnitude.

The reason is that treating the NAS as if it were local means that the receiving rsync (which is still running on the client machine!) is reading huge quantities of data in order to check for modified files. Doing it the other way means the receiver runs on the NAS and the network is only used to send summaries and changed files, with the resulting boost in efficiency. It also means you don&#39;t need NFS for backups (though it&#39;s still useful for restores).

[/quote]

Hi Patrick, I&#39;m not sure what you mean by "client" in this scenario. My setup is as follows. I have a QNAP NAS where all our important files are stored. In addition I have a D-Link NAS which I&#39;m using as the target for backups of the QNAP NAS, so if the QNAP blows up we have a backup copy of everything on the D-Link. 

I&#39;m running rsnapshot on the QNAP NAS which backs up the data onto the D-LInk NAS over NFS. I&#39;m not sure which of these would be the client in your terminology. But I have noticed that the backups from the QNAP to the D-Link are quite slow so any suggestions on how to speed them up are welcome. 

On 19 May 2016 at 10:22, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote][quote]
I haven&#39;t really been following this thread but I have one question: are you running rsnapshot on the NAS or on the client(s)? The fact that you mount the NAS via NFS leads me to suspect the former. It&#39;s worth noting that this is by far the worst (i.e. least efficient) way to do it. I used to do it this way a few years ago because it seemed so natural, but in fact rsnapshot works much better if you run it on the NAS and connect to the client(s) via the rsync protocol. This is not very clear from the documentation - I guess the author just assumed it was obvious - but when I changed methods I saw a speedup by an order of magnitude.

The reason is that treating the NAS as if it were local means that the receiving rsync (which is still running on the client machine!) is reading huge quantities of data in order to check for modified files. Doing it the other way means the receiver runs on the NAS and the network is only used to send summaries and changed files, with the resulting boost in efficiency. It also means you don&#39;t need NFS for backups (though it&#39;s still useful for restores).
[/quote]

Hi Patrick, I&#39;m not sure what you mean by "client" in this scenario. My setup is as follows. I have a QNAP NAS where all our important files are stored. In addition I have a D-Link NAS which I&#39;m using as the target for backups of the QNAP NAS, so if the QNAP blows up we have a backup copy of everything on the D-Link.

I&#39;m running rsnapshot on the QNAP NAS which backs up the data onto the D-LInk NAS over NFS. I&#39;m not sure which of these would be the client in your terminology. But I have noticed that the backups from the QNAP to the D-Link are quite slow so any suggestions on how to speed them up are welcome.
[/quote]

In this context client means "machine whose files are being backed up" and server is "machine where the backups are stored". So in your case the QNAP is the client and the D-Link is the server.

From what you say, running rsnapshot on the QNAP is what you don&#39;t want to do. You should be running it on the D-Link. This takes a little more work to set up but is very much worth it in terms of efficiency. You need to configure the QNAP to allow rsync access. If you&#39;re concerned about security, best do this as rsync over ssh, which means first of all setting up ssh login from the D-Link to the QNAP. As you want to do this as a batch process, the login should use token authentication, not passwords. Make sure that any intervening firewalls (e.g. on the QNAP itself) will let ssh through.

After that, it&#39;s a question of configuring your rsnapshot.conf file (on the *server* side) to use rsync. As an example, I have lines like these (where Bree is my client machine):

backup  Bree::backup-home       ./home
backup  Bree::backup-etc        ./etc
backup  Bree::backup-local      ./local
backup  Bree::backup-opt        ./opt
...(note that this is slightly different from the default, as I use a separate rsnapshot root for each possible client).

And on Bree itself, I have these lines in /etc/rsyncd.conf:

[backup-home]
       path=/home

[backup-etc]
       path=/etc

[backup-local]
       path=/usr/local

[backup-opt]
       path=/opt
...

The main headache getting this all to work was actually with ssh. For some reason it kept losing the authentication token so rsnapshot would fail. I finally figured out that the problem was with my DNS setup. The server wasn&#39;t resolving Bree&#39;s name correctly :-)

Anyway, best of luck.

poc

[quote]In this context client means "machine whose files are being backed up" and server is "machine where the backups are stored". So in your case the QNAP is the client and the D-Link is the server.

From what you say, running rsnapshot on the QNAP is what you don&#39;t want to do. You should be running it on the D-Link. This takes a little more work to set up but is very much worth it in terms of efficiency. You need to configure the QNAP to allow rsync access. If you&#39;re concerned about security, best do this as rsync over ssh, which means first of all setting up ssh login from the D-Link to the QNAP. As you want to do this as a batch process, the login should use token authentication, not passwords. Make sure that any intervening firewalls (e.g. on the QNAP itself) will let ssh through.

After that, it&#39;s a question of configuring your rsnapshot.conf file (on the *server* side) to use rsync. As an example, I have lines like these (where Bree is my client machine):

The main headache getting this all to work was actually with ssh. For some reason it kept losing the authentication token so rsnapshot would fail. I finally figured out that the problem was with my DNS setup. The server wasn&#39;t resolving Bree&#39;s name correctly :-)

[/quote]

Thanks very much for this advice, Patrick. I&#39;ll give it a try. Hopefully the limited linux O/S on my QNAP NAS won&#39;t prevent it from working.

Out of interest, any idea why "pulling" it from the client to the server should be so much faster than "pushing" it from the client?

On 19 May 2016 at 12:43, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote]Thanks very much for this advice, Patrick. I&#39;ll give it a try. Hopefully the limited linux O/S on my QNAP NAS won&#39;t prevent it from working.

Out of interest, any idea why "pulling" it from the client to the server should be so much faster than "pushing" it from the client?
[/quote]

As I said in an earlier message, it&#39;s because of how rsync works. There are two rsync processes, one at each end of the connection. Call them the source and the target processes, or S and T for short. On startup S sends a list of candidate files to T. T checks which of them it doesn&#39;t already have and asks S to send them. It also checks which of the files it already has may have changed, firstly by looking at sizes and then at checksums. The exact behaviour is controlled by rsync options but we&#39;ll take the default case. When a file exists at both ends but is different, there is a rolling checksum that tries to minimize the number of sections of file to re-send in order to bring the T version up to date. (This is a very crude summary but it&#39;s enough for now).

OK, what happens when we&#39;re doing this on NFS? Both S and T are running *on the same machine* (the client). S is looking at local files and T at NFS-mounted remote files, so for T to do any checksumming it has to read the file contents over NFS. This is a big performance hit, which you have to pay *even when the files are not actually different*. In other words, in the NFS scenario the cost of rsnapshot is dominated not by the number of changed files, but by the total size of the files you want to check.

On the other hand, using the rsync "pull" method, T and S are on different machines (so potentially faster for that reason) and only files that actually might be different will be copied. Big win.

Cheers

poc

[quote][quote]From what you say, running rsnapshot on the QNAP is what you don&#39;t want to do. You should be running it on the D-Link. This takes a little more work to set up but is very much worth it in terms of efficiency. You need to configure the QNAP to allow rsync access. If you&#39;re concerned about security, best do this as rsync over ssh, which means first of all setting up ssh login from the D-Link to the QNAP. As you want to do this as a batch process, the login should use token authentication, not passwords. Make sure that any intervening firewalls (e.g. on the QNAP itself) will let ssh through.

After that, it&#39;s a question of configuring your rsnapshot.conf file (on the *server* side) to use rsync. 

[/quote]

[/quote]

I don&#39;t know if I set something up incorrectly here, by my experience of this was totally different to your&#39;s. I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot and "pulls" the files from the QNAP NAS over a CIFS connection. Unfortunately this was massively slower than my original setup - more than 3 times slower. I took some timings:

New method      - D-Link NAS pulls from QNAP over CIFS       - 12.3GB home directory tree for one user - 145 minutes.
Original method - QNAP NAS pushes to D-Link NAS over NFS - 12.3GB home directory tree for one user - 44 minutes.

As you can see from these timings, mounting the remote D-Link NAS over NFS and pushing files to it from the QNAP was a LOT quicker. 

As I say, I don&#39;t know whether I set something up incorrectly, but I don&#39;t think so. Interestingly the QNAP TS-410 has a slower CPU (ARM Marvell 800MHz) than the D-Link DNS-320L (ARM Marvell 1000MHz) so you would have thought it would run rsnapshot/rsync slower. They both have 256MB RAM.

On 22 May 2016 at 13:56, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote]I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot and "pulls" the files from the QNAP NAS over a CIFS connection[/quote]

I suspect you didn&#39;t entirely understand the reasoning behind this. Who said anything about CIFS? That&#39;s just mounting the filesystems in reverse. I&#39;m not surprised it didn&#39;t work well. The whole point is to back up using the rsync protocol *and not a remote filesystem*.

poc

On 22 May 2016 23:01, "Patrick O&#39;Callaghan" <pocallaghan < at > gmail.com ([email]pocallaghan < at > gmail.com[/email])> wrote:
[quote]

On 22 May 2016 at 13:56, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote]
I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot and "pulls" the files from the QNAP NAS over a CIFS connection
[/quote]

I suspect you didn&#39;t entirely understand the reasoning behind this. Who said anything about CIFS? That&#39;s just mounting the filesystems in reverse. I&#39;m not surprised it didn&#39;t work well. The whole point is to back up using the rsync protocol *and not a remote filesystem*.

[/quote]Sorry, I muddled up SSH and CIFS. The remote D-Link NAS connected to the QNAP by SSH. There was no CIFS involved. So that means rsync was running on the QNAP, and maybe it ran very slowly?

On 22 May 2016 at 22:04, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote]
On 22 May 2016 23:01, "Patrick O&#39;Callaghan" <pocallaghan < at > gmail.com ([email]pocallaghan < at > gmail.com[/email])> wrote:
[quote]

On 22 May 2016 at 13:56, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote]
I changed my rsnapshot backup so that the D-Link NAS2 runs rsnapshot and "pulls" the files from the QNAP NAS over a CIFS connection
[/quote]

I suspect you didn&#39;t entirely understand the reasoning behind this. Who said anything about CIFS? That&#39;s just mounting the filesystems in reverse. I&#39;m not surprised it didn&#39;t work well. The whole point is to back up using the rsync protocol *and not a remote filesystem*.

[/quote]Sorry, I muddled up SSH and CIFS. The remote D-Link NAS connected to the QNAP by SSH. There was no CIFS involved. So that means rsync was running on the QNAP, and maybe it ran very slowly?[/quote]

In fact rsync is running on both machines. That&#39;s how it works. My advice was to run rsnapshot on the backup server and configure it to run rsync (optionally over ssh) to the client. It does this by running rsync at both ends, the two processes cooperating to minimize transfers.

It&#39;s possible that the backup server (i.e. the D-Link in your case) is just a slow machine and the ssh processing is taking too long (unlikely), or perhaps it has slow disk drives. That could happen if its disks are on USB 2 ports for example, but of course the same would apply in your original configuration. So I don&#39;t really understand what&#39;s happening.

In my case the backup server is an Iomega NAS, around 5 years old now, with internal drives in a raid configuration. It&#39;s certainly not a fast machine and as I said earlier I saw an order of magnitude speedup by changing rsnapshot to pull with rsync rather than push over NFS.

poc

[quote]In fact rsync is running on both machines. That&#39;s how it works. My advice was to run rsnapshot on the backup server and configure it to run rsync (optionally over ssh) to the client. It does this by running rsync at both ends, the two processes cooperating to minimize transfers.

It&#39;s possible that the backup server (i.e. the D-Link in your case) is just a slow machine and the ssh processing is taking too long (unlikely), or perhaps it has slow disk drives. That could happen if its disks are on USB 2 ports for example, but of course the same would apply in your original configuration. So I don&#39;t really understand what&#39;s happening.

In my case the backup server is an Iomega NAS, around 5 years old now, with internal drives in a raid configuration. It&#39;s certainly not a fast machine and as I said earlier I saw an order of magnitude speedup by changing rsnapshot to pull with rsync rather than push over NFS.

[/quote]

Thanks Patrick. I guess for anyone looking at this in future the recommendation would be to try both options and see what works best for their particular setup.

Appreciate all your help on this and it&#39;s been a good learning exercise for me.

On Sun, May 22, 2016 at 9:41 PM, Phill Edwards <philledwards < at > gmail.com> wrote:
[quote][quote]In fact rsync is running on both machines. That's how it works. My advice
was to run rsnapshot on the backup server and configure it to run rsync
(optionally over ssh) to the client. It does this by running rsync at both
ends, the two processes cooperating to minimize transfers.

It's possible that the backup server (i.e. the D-Link in your case) is
just a slow machine and the ssh processing is taking too long (unlikely), or
perhaps it has slow disk drives. That could happen if its disks are on USB 2
ports for example, but of course the same would apply in your original
configuration. So I don't really understand what's happening.

In my case the backup server is an Iomega NAS, around 5 years old now,
with internal drives in a raid configuration. It's certainly not a fast
machine and as I said earlier I saw an order of magnitude speedup by
changing rsnapshot to pull with rsync rather than push over NFS.
[/quote]

Thanks Patrick. I guess for anyone looking at this in future the
recommendation would be to try both options and see what works best for
their particular setup.
[/quote]
Synology servers are rataher limited, proprietized environment. The
one I've looked at recently does not even have "perl" available to the
running operating system. It uses an ext4 backend filesystem, which is
very stable, but does have limits with excessive numbers of files in
single directories or filesystems that are too deep. It's LVM mounted,
but that allows only one LVM snapshot maximum, not multiple local
filesystem snapshots.

If one must do rsnapshot backups to a Synology server for reasons of
bulk storage and accessible backups, I'd strongly suggest using NFS
mounts, and limiting the number of "hourly" or most frequently
performed backups to control the burden of the hardlinks. NFS is much,
much less chatty than CIFS, and is likely to perform far better. It
almost always does for any production environment. The big advantage
of CIFS is the availability of the complex permission models only
available to NFSv4 for NFS environments, and those are not normally
that useful.

[quote]Appreciate all your help on this and it's been a good learning exercise for
me.

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

[/quote]
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
rsnapshot-discuss mailing list
rsnapshot-discuss < at > lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

On 23 May 2016 at 02:41, Phill Edwards <philledwards < at > gmail.com ([email]philledwards < at > gmail.com[/email])> wrote:
[quote][quote]In fact rsync is running on both machines. That&#39;s how it works. My advice was to run rsnapshot on the backup server and configure it to run rsync (optionally over ssh) to the client. It does this by running rsync at both ends, the two processes cooperating to minimize transfers.

It&#39;s possible that the backup server (i.e. the D-Link in your case) is just a slow machine and the ssh processing is taking too long (unlikely), or perhaps it has slow disk drives. That could happen if its disks are on USB 2 ports for example, but of course the same would apply in your original configuration. So I don&#39;t really understand what&#39;s happening.

In my case the backup server is an Iomega NAS, around 5 years old now, with internal drives in a raid configuration. It&#39;s certainly not a fast machine and as I said earlier I saw an order of magnitude speedup by changing rsnapshot to pull with rsync rather than push over NFS.

[/quote]

Thanks Patrick. I guess for anyone looking at this in future the recommendation would be to try both options and see what works best for their particular setup.

Appreciate all your help on this and it&#39;s been a good learning exercise for me.
[/quote]

Happy to help, even though I didn&#39;t in the end :-)

Cheers

poc