Welcome! » Log In » Create A New Profile

corrupted files in destination while source is good and no errors when running rdiff-backup

Posted by Jelle de Jong 
Hello everybody,

Thank you for rdiff-backup it has been very useful for me for many years.

I am making a new solution to be able to use rdiff-backup with ntfs3
with a new dedup plugin, this seem to be working now.

I ran rdiff-backup on the same volume, some of these runs have been done
on currupt files while fine tuning the de-duplication read plugin.

root@backup:~# rdiff-backup --list-increments
/srv/storage/backups/libvirt-filesystems/sr7-sdb2/
Found 9 increments:
increments.2017-02-09T18:52:53+01:00.dir Thu Feb 9 18:52:53 2017
increments.2017-02-12T03:57:18+01:00.dir Sun Feb 12 03:57:18 2017
increments.2017-02-24T12:42:23+01:00.dir Fri Feb 24 12:42:23 2017
increments.2017-02-27T13:45:11+01:00.dir Mon Feb 27 13:45:11 2017
increments.2017-02-27T17:03:21+01:00.dir Mon Feb 27 17:03:21 2017
increments.2017-02-28T14:56:43+01:00.dir Tue Feb 28 14:56:43 2017
increments.2017-03-05T04:40:29+01:00.dir Sun Mar 5 04:40:29 2017
increments.2017-03-09T19:24:12+01:00.dir Thu Mar 9 19:24:12 2017
increments.2017-03-12T03:30:37+01:00.dir Sun Mar 12 03:30:37 2017
Current mirror: Tue Mar 14 19:26:57 2017

We fixed all the errors and I can read all the documents from the source
directory fine , but rdiff is having corrupt documents in the backup
destination.

When testing I found documents with different md5sums in my backup and
the source destination.

After some investigating --verify outputs the bad documents:

http://paste.debian.net/hidden/d311cfcf/

Why is rdiff-backup running fine over the directory but still has
corrupted documents in the destination and how do I fix this without
removing the full backup.

Kind regards,

Jelle de Jong

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
This message was imported via the External PhorumMail Module
This is just a guess, but if your de-duplication process involves the
use of hard links which rdiff-backup is backing up, then your problem
may relate to the following bug (which affects hard links):

http://savannah.nongnu.org/bugs/?26848

The bug can generate the type of errors shown in your verify log.

See if your version of rdiff-backup includes the patches attached to the
bug report. If not, apply the patches and see if that fixes any of your
problems.

--Joe


On 3/16/2017 9:43 AM, Jelle de Jong wrote:
> Hello everybody,
>
> Thank you for rdiff-backup it has been very useful for me for many years.
>
> I am making a new solution to be able to use rdiff-backup with ntfs3
> with a new dedup plugin, this seem to be working now.
>
> I ran rdiff-backup on the same volume, some of these runs have been done
> on currupt files while fine tuning the de-duplication read plugin.
>
> root@backup:~# rdiff-backup --list-increments
> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/
> Found 9 increments:
> increments.2017-02-09T18:52:53+01:00.dir Thu Feb 9 18:52:53 2017
> increments.2017-02-12T03:57:18+01:00.dir Sun Feb 12 03:57:18 2017
> increments.2017-02-24T12:42:23+01:00.dir Fri Feb 24 12:42:23 2017
> increments.2017-02-27T13:45:11+01:00.dir Mon Feb 27 13:45:11 2017
> increments.2017-02-27T17:03:21+01:00.dir Mon Feb 27 17:03:21 2017
> increments.2017-02-28T14:56:43+01:00.dir Tue Feb 28 14:56:43 2017
> increments.2017-03-05T04:40:29+01:00.dir Sun Mar 5 04:40:29 2017
> increments.2017-03-09T19:24:12+01:00.dir Thu Mar 9 19:24:12 2017
> increments.2017-03-12T03:30:37+01:00.dir Sun Mar 12 03:30:37 2017
> Current mirror: Tue Mar 14 19:26:57 2017
>
> We fixed all the errors and I can read all the documents from the source
> directory fine , but rdiff is having corrupt documents in the backup
> destination.
>
> When testing I found documents with different md5sums in my backup and
> the source destination.
>
> After some investigating --verify outputs the bad documents:
>
> http://paste.debian.net/hidden/d311cfcf/
>
> Why is rdiff-backup running fine over the directory but still has
> corrupted documents in the destination and how do I fix this without
> removing the full backup.
>
> Kind regards,
>
> Jelle de Jong
>
> _______________________________________________
> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>
>

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
This message was imported via the External PhorumMail Module
The files where no hard links, on the ntfs source directory they show up
as real files.

Kind regards,

Jelle de Jong


On 16/03/17 19:12, Joe Steele wrote:
> This is just a guess, but if your de-duplication process involves the
> use of hard links which rdiff-backup is backing up, then your problem
> may relate to the following bug (which affects hard links):
>
> http://savannah.nongnu.org/bugs/?26848
>
> The bug can generate the type of errors shown in your verify log.
>
> See if your version of rdiff-backup includes the patches attached to the
> bug report. If not, apply the patches and see if that fixes any of your
> problems.
>
> --Joe
>
>
> On 3/16/2017 9:43 AM, Jelle de Jong wrote:
>> Hello everybody,
>>
>> Thank you for rdiff-backup it has been very useful for me for many years.
>>
>> I am making a new solution to be able to use rdiff-backup with ntfs3
>> with a new dedup plugin, this seem to be working now.
>>
>> I ran rdiff-backup on the same volume, some of these runs have been done
>> on currupt files while fine tuning the de-duplication read plugin.
>>
>> root@backup:~# rdiff-backup --list-increments
>> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/
>> Found 9 increments:
>> increments.2017-02-09T18:52:53+01:00.dir Thu Feb 9 18:52:53 2017
>> increments.2017-02-12T03:57:18+01:00.dir Sun Feb 12 03:57:18 2017
>> increments.2017-02-24T12:42:23+01:00.dir Fri Feb 24 12:42:23 2017
>> increments.2017-02-27T13:45:11+01:00.dir Mon Feb 27 13:45:11 2017
>> increments.2017-02-27T17:03:21+01:00.dir Mon Feb 27 17:03:21 2017
>> increments.2017-02-28T14:56:43+01:00.dir Tue Feb 28 14:56:43 2017
>> increments.2017-03-05T04:40:29+01:00.dir Sun Mar 5 04:40:29 2017
>> increments.2017-03-09T19:24:12+01:00.dir Thu Mar 9 19:24:12 2017
>> increments.2017-03-12T03:30:37+01:00.dir Sun Mar 12 03:30:37 2017
>> Current mirror: Tue Mar 14 19:26:57 2017
>>
>> We fixed all the errors and I can read all the documents from the source
>> directory fine , but rdiff is having corrupt documents in the backup
>> destination.
>>
>> When testing I found documents with different md5sums in my backup and
>> the source destination.
>>
>> After some investigating --verify outputs the bad documents:
>>
>> http://paste.debian.net/hidden/d311cfcf/
>>
>> Why is rdiff-backup running fine over the directory but still has
>> corrupted documents in the destination and how do I fix this without
>> removing the full backup.
>>
>> Kind regards,
>>
>> Jelle de Jong
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL:
>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>
>>

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
This message was imported via the External PhorumMail Module
root@backup:~# md5sum
/srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
8cbb1482783c05a1604987532bdc6540
/srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
root@backup:~# md5sum /mnt/sr7-sdb2/<snip>/DSC_1319.JPG
913ee9c4832f69cc60a99dbc7b9e57c9 /mnt/sr7-sdb2/<snip>/DSC_1319.JPG

root@backup:~# ls -hal
/srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
-r-xr-xr-x 1 root root 2.4M Feb 5 2014
/srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
root@backup:~# ls -hal /mnt/sr7-sdb2/<snip>/DSC_1319.JPG
-r-xr-xr-x 1 root root 2.4M Feb 5 2014 /mnt/sr7-sdb2/<snip>/DSC_1319.JPG


Kind regards,

Jelle de Jong


On 16/03/17 19:12, Joe Steele wrote:
> This is just a guess, but if your de-duplication process involves the
> use of hard links which rdiff-backup is backing up, then your problem
> may relate to the following bug (which affects hard links):
>
> http://savannah.nongnu.org/bugs/?26848
>
> The bug can generate the type of errors shown in your verify log.
>
> See if your version of rdiff-backup includes the patches attached to the
> bug report. If not, apply the patches and see if that fixes any of your
> problems.
>
> --Joe
>
>
> On 3/16/2017 9:43 AM, Jelle de Jong wrote:
>> Hello everybody,
>>
>> Thank you for rdiff-backup it has been very useful for me for many years.
>>
>> I am making a new solution to be able to use rdiff-backup with ntfs3
>> with a new dedup plugin, this seem to be working now.
>>
>> I ran rdiff-backup on the same volume, some of these runs have been done
>> on currupt files while fine tuning the de-duplication read plugin.
>>
>> root@backup:~# rdiff-backup --list-increments
>> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/
>> Found 9 increments:
>> increments.2017-02-09T18:52:53+01:00.dir Thu Feb 9 18:52:53 2017
>> increments.2017-02-12T03:57:18+01:00.dir Sun Feb 12 03:57:18 2017
>> increments.2017-02-24T12:42:23+01:00.dir Fri Feb 24 12:42:23 2017
>> increments.2017-02-27T13:45:11+01:00.dir Mon Feb 27 13:45:11 2017
>> increments.2017-02-27T17:03:21+01:00.dir Mon Feb 27 17:03:21 2017
>> increments.2017-02-28T14:56:43+01:00.dir Tue Feb 28 14:56:43 2017
>> increments.2017-03-05T04:40:29+01:00.dir Sun Mar 5 04:40:29 2017
>> increments.2017-03-09T19:24:12+01:00.dir Thu Mar 9 19:24:12 2017
>> increments.2017-03-12T03:30:37+01:00.dir Sun Mar 12 03:30:37 2017
>> Current mirror: Tue Mar 14 19:26:57 2017
>>
>> We fixed all the errors and I can read all the documents from the source
>> directory fine , but rdiff is having corrupt documents in the backup
>> destination.
>>
>> When testing I found documents with different md5sums in my backup and
>> the source destination.
>>
>> After some investigating --verify outputs the bad documents:
>>
>> http://paste.debian.net/hidden/d311cfcf/
>>
>> Why is rdiff-backup running fine over the directory but still has
>> corrupted documents in the destination and how do I fix this without
>> removing the full backup.
>>
>> Kind regards,
>>
>> Jelle de Jong
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL:
>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>
>>

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL:
http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
This message was imported via the External PhorumMail Module
Anybody?

I will remove the whole backup and start over again, but I would love
some advice on how to have fixed the current backup (I saved the backups
for debugging).

Kind regards,

Jelle de Jong

On 17/03/17 12:54, Jelle de Jong wrote:
> root@backup:~# md5sum
> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
> 8cbb1482783c05a1604987532bdc6540
> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
> root@backup:~# md5sum /mnt/sr7-sdb2/<snip>/DSC_1319.JPG
> 913ee9c4832f69cc60a99dbc7b9e57c9 /mnt/sr7-sdb2/<snip>/DSC_1319.JPG
>
> root@backup:~# ls -hal
> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
> -r-xr-xr-x 1 root root 2.4M Feb 5 2014
> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/<snip>/DSC_1319.JPG
> root@backup:~# ls -hal /mnt/sr7-sdb2/<snip>/DSC_1319.JPG
> -r-xr-xr-x 1 root root 2.4M Feb 5 2014 /mnt/sr7-sdb2/<snip>/DSC_1319.JPG
>
>
> Kind regards,
>
> Jelle de Jong
>
>
> On 16/03/17 19:12, Joe Steele wrote:
>> This is just a guess, but if your de-duplication process involves the
>> use of hard links which rdiff-backup is backing up, then your problem
>> may relate to the following bug (which affects hard links):
>>
>> http://savannah.nongnu.org/bugs/?26848
>>
>> The bug can generate the type of errors shown in your verify log.
>>
>> See if your version of rdiff-backup includes the patches attached to the
>> bug report. If not, apply the patches and see if that fixes any of your
>> problems.
>>
>> --Joe
>>
>>
>> On 3/16/2017 9:43 AM, Jelle de Jong wrote:
>>> Hello everybody,
>>>
>>> Thank you for rdiff-backup it has been very useful for me for many
>>> years.
>>>
>>> I am making a new solution to be able to use rdiff-backup with ntfs3
>>> with a new dedup plugin, this seem to be working now.
>>>
>>> I ran rdiff-backup on the same volume, some of these runs have been done
>>> on currupt files while fine tuning the de-duplication read plugin.
>>>
>>> root@backup:~# rdiff-backup --list-increments
>>> /srv/storage/backups/libvirt-filesystems/sr7-sdb2/
>>> Found 9 increments:
>>> increments.2017-02-09T18:52:53+01:00.dir Thu Feb 9 18:52:53 2017
>>> increments.2017-02-12T03:57:18+01:00.dir Sun Feb 12 03:57:18 2017
>>> increments.2017-02-24T12:42:23+01:00.dir Fri Feb 24 12:42:23 2017
>>> increments.2017-02-27T13:45:11+01:00.dir Mon Feb 27 13:45:11 2017
>>> increments.2017-02-27T17:03:21+01:00.dir Mon Feb 27 17:03:21 2017
>>> increments.2017-02-28T14:56:43+01:00.dir Tue Feb 28 14:56:43 2017
>>> increments.2017-03-05T04:40:29+01:00.dir Sun Mar 5 04:40:29 2017
>>> increments.2017-03-09T19:24:12+01:00.dir Thu Mar 9 19:24:12 2017
>>> increments.2017-03-12T03:30:37+01:00.dir Sun Mar 12 03:30:37 2017
>>> Current mirror: Tue Mar 14 19:26:57 2017
>>>
>>> We fixed all the errors and I can read all the documents from the source
>>> directory fine , but rdiff is having corrupt documents in the backup
>>> destination.
>>>
>>> When testing I found documents with different md5sums in my backup and
>>> the source destination.
>>>
>>> After some investigating --verify outputs the bad documents:
>>>
>>> http://paste.debian.net/hidden/d311cfcf/
>>>
>>> Why is rdiff-backup running fine over the directory but still has
>>> corrupted documents in the destination and how do I fix this without
>>> removing the full backup.
>>>
>>> Kind regards,
>>>
>>> Jelle de Jong
>>>
>>> _______________________________________________
>>> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
>>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>>> Wiki URL:
>>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>>>
>>>
>
> _______________________________________________
> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>
> _______________________________________________
> rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
This message was imported via the External PhorumMail Module
Sorry, only registered users may post in this forum.

Click here to login