Backup Central

Hi folks,

I'm abusing rdiff-backup as a "write from one" + "read from many"
solution.

Only one host backs up to the central server, as the user with
permission to write files.

Many hosts will restore from that central server, as a user who does not
have access to write the files.

I touch up permissions after backing up, making the metadata files o+r
and the metadata directories o+rx, to facilitate this.

The restore command looks like this:

TZ=UTC rdiff-backup \
--restore-as-of "2005-07-18" \
--force \
--exclude /usr/portage/distfiles \
fatboy < at > backupserver.clue.co.za /usr/portage

However, the restores exception out on "Operation not permitted", as per
the attached trace.

Since the local user on the "restoree" host's end is root, I can only
assume that the problem is on the "backup server" (remote) end.
However, I'd have expected a restore to be a read-only operation
(except for the failure to write backup.log).

Am I doing something wrong, or is this just outside the design of
rdiff-backup? :-(

Ciao,
Sheldon.

OSError: [Errno 1] Operation not permitted: 'portage-rdiff-backup/app-accessibility'
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 254, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 224, in take_action
connection.PipeConnection(sys.stdin, sys.stdout).Server()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 347, in Server
self.get_response(-1)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 309, in get_response
try: req_num, object = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 225, in _get
raise ConnectionReadError("Truncated header string (problem "
rdiff_backup.connection.ConnectionReadError: Truncated header string (problem probably originated remotely)

On Mon, 18 Jul 2005, Sheldon Hearn wrote:

However, the restores exception out on "Operation not permitted", as per
the attached trace.

could you do this with -v5? i'm always skeptical of the traces that are
reported when remote is involved...

-dean

On Monday 18 July 2005 22:49, dean gaudet wrote:
However, the restores exception out on "Operation not permitted",
as per the attached trace.

could you do this with -v5?  i'm always skeptical of the traces that
are reported when remote is involved...

Sure, thanks for taking an interest.

The warning about being unable to write to the log is included for
completeness, but I think it's a non-issue, since it's only a warning
and it shouldn't matter for a restore. It's the "Operation not
permitted" that's getting in my way.

There's nothing special about the packages directory, by the way.
yesterday, it was a different directory. It seems to be the first
directory for which the restore operation would actually do something
to the local target directory.

Ciao,
Sheldon.

Well, I think that exactly matters. Try again permitting writing to the log
and report again...

Am Dienstag 19 Juli 2005 17:15 schrieb Sheldon Hearn:
On Monday 18 July 2005 22:49, dean gaudet wrote:
However, the restores exception out on "Operation not permitted",
as per the attached trace.

could you do this with -v5?  i'm always skeptical of the traces that
are reported when remote is involved...

Sure, thanks for taking an interest.

The warning about being unable to write to the log is included for
completeness, but I think it's a non-issue, since it's only a warning
and it shouldn't matter for a restore. It's the "Operation not
permitted" that's getting in my way.

There's nothing special about the packages directory, by the way.
yesterday, it was a different directory. It seems to be the first
directory for which the restore operation would actually do something
to the local target directory.

Ciao,
Sheldon.

--
Vorteile von OpenOffice:
- GRATIS
- einfach zu bedienen
- kompatibel
- portabel
Infos: http://de.openoffice.org/marketing/collaterals/faltflyer_11.pdf

Ich schreibe alle Arbeiten damit.

that sure looks like it's trying to chmod something on the mirror doesn't
it:

Sending back exception [Errno 1] Operation not permitted: 'portage-rdiff-backup/packages' of type exceptions.OSError:
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 329, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 475, in readfromid
if length is None: return cls.vfiles[id].read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 285, in read
if not self.addtobuffer(): break
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 308, in addtobuffer
try: currentobj = self.iter.next()
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 244, in get_diffs_from_collated
yield cls.get_diff(mir_rorp, target_rorp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 253, in get_diff
mir_rorp.setfile(cls.rf_cache.get_fp(expanded_index))
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 339, in get_fp
rf = self.get_rf(index)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 324, in get_rf
if not self.add_rfs(index): return None
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 356, in add_rfs
if Globals.process_uid != 0: self.perm_changer(parent_index)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 675, in __call__
self.add_new(old_index, index)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 693, in add_new
else: rp.chmod(0700 | old_perms)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 629, in chmod
self.conn.os.chmod(self.path, permissions)

that's not good...

btw is portage-rdiff-backup/packages readable by the user you're using
to restore?

-dean

Why do you think the failure to modify the restore log makes a
difference?

As you requested, I made it writable, and still get "Operation not
permitted" on an unrelated object (currently the packages
subdirectory).

Fresh trace included.

Anyone actually know what rdiff-backup thinks its doing here? If I
could find the wayward chmod/chown in the source, I'd just hack it out,
since it's bogus for my application, and probably bogus in any restore.

Ciao,
Sheldon.

On Tuesday 19 July 2005 17:30, Thomas Bettler wrote:
Well, I think that exactly matters. Try again permitting writing to
the log and report again...

Am Dienstag 19 Juli 2005 17:15 schrieb Sheldon Hearn:
On Monday 18 July 2005 22:49, dean gaudet wrote:
However, the restores exception out on "Operation not
permitted", as per the attached trace.

could you do this with -v5?  i'm always skeptical of the traces
that are reported when remote is involved...

Sure, thanks for taking an interest.

The warning about being unable to write to the log is included for
completeness, but I think it's a non-issue, since it's only a
warning and it shouldn't matter for a restore. It's the "Operation
not permitted" that's getting in my way.

There's nothing special about the packages directory, by the way.
yesterday, it was a different directory. It seems to be the first
directory for which the restore operation would actually do
something to the local target directory.

Ciao,
Sheldon.

Script started on Wed Jul 20 09:26:27 2005
Using Fatboy release date: 2005-07-18
Executing ssh -C fatboy < at > urchin.clue.co.za rdiff-backup --server
Starting restore of portage-rdiff-backup to /usr/portage as it was as of Mon Jul 18 00:00:00 2005.
Sending back exception [Errno 1] Operation not permitted: 'portage-rdiff-backup/packages' of type exceptions.OSError:
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 329, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 475, in readfromid
if length is None: return cls.vfiles[id].read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 285, in read
if not self.addtobuffer(): break
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 308, in addtobuffer
try: currentobj = self.iter.next()
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 244, in get_diffs_from_collated
yield cls.get_diff(mir_rorp, target_rorp)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 253, in get_diff
mir_rorp.setfile(cls.rf_cache.get_fp(expanded_index))
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 339, in get_fp
rf = self.get_rf(index)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 324, in get_rf
if not self.add_rfs(index): return None
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 356, in add_rfs
if Globals.process_uid != 0: self.perm_changer(parent_index)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 675, in __call__
self.add_new(old_index, index)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 693, in add_new
else: rp.chmod(0700 | old_perms)
File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 629, in chmod
self.conn.os.chmod(self.path, permissions)

Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 23, in ?
rdiff_backup.Main.Main(sys.argv[1:])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 254, in Main
take_action(rps)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 228, in take_action
elif action == "restore-as-of": RestoreAsOf(rps[0], rps[1])
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 401, in RestoreAsOf
restore_common(rpin, target, time)
File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 413, in restore_common
restore.Restore(mirror, inc_rpath, target, time)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 45, in Restore
TargetS.patch(target, diff_iter)
File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 283, in patch
for diff in rorpiter.FillInIter(diff_iter, target):
File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 173, in FillInIter
first_rp = rpiter.next() # StopIteration gets passed upwards
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 364, in next
while not type: type, data = self._get()
File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 401, in _get
if not self.buf: self.buf += self.file.read()
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 508, in read
return self.connection.VirtualFile.readfromid(self.id, length)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 440, in __call__
return apply(self.connection.reval, (self.name,) + args)
File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", line 362, in reval
if isinstance(result, Exception): raise result
OSError: [Errno 1] Operation not permitted: 'portage-rdiff-backup/packages'
Traceback (most recent call last):

Script done on Wed Jul 20 09:27:21 2005

On Wednesday 20 July 2005 11:39, Sheldon Hearn wrote:
Anyone actually know what rdiff-backup thinks its doing here?  If I
could find the wayward chmod/chown in the source, I'd just hack it
out, since it's bogus for my application, and probably bogus in any
restore.

I've taken a look at the code in restore.py, and I don't get it.

If we're not root on the server during a restore operation, we try to
change permissions to ensure readability? Either we can read an object
or we can't. If we can't, it's _highly_ unlikely that we'll be able to
change permissions on it anyway.

The attached patch fixes rdiff-backup for my admittedly unusual
application. Be warned, I have no idea why the code I removed was put
there in the first place. Just because it doesn't make sense to me
doesn't mean it had no purpose.

So I'll just carry this patch on my central server (it's not required on
restore clients), and we can all pretend this didn't happen until the
next person tries to use rdiff-backup to restore from a read-only
location. :-)

Ciao,
Sheldon.

Hi,

On Wed, 20 Jul 2005, Sheldon Hearn wrote:

As you requested, I made it writable, and still get "Operation not
permitted" on an unrelated object (currently the packages
subdirectory).

Without looking at the trace or at the source, I'd say it may make sense
in some situations. Like: having stuff below a 'closed' directory. When
backing up, permissions are temporarily changed to allow writes, and my
guess is that the same thing happens when restoring.

That might be related to your problem.

Off-topic: can't you use plain rsync for this task?

Regards,
Maarten

On Wednesday 20 July 2005 13:26, Maarten Bezemer wrote:
Without looking at the trace or at the source, I'd say it may make
sense in some situations. Like: having stuff below a 'closed'
directory. When backing up, permissions are temporarily changed to
allow writes, and my guess is that the same thing happens when
restoring.

That might be related to your problem.

Off-topic: can't you use plain rsync for this task?

Yes. If the maintainers are reluctant to address this in rdiff-backup,
I'll use cpio+hardlink backups on the distribution server and rsync to
pull snapshots down on the clients.

Ciao,
Sheldon.
--
Sheldon Hearn
IT Director
Clue Technologies (PTY) Ltd

Web: http://www.clue.co.za/
Mail: sheldonh < at > clue.co.za
Office: +27-21-434-8034
Mobile: +27-83-564-3276
Timezone: SAST (+0200)

On Wed, 20 Jul 2005, Sheldon Hearn wrote:

If we're not root on the server during a restore operation, we try to
change permissions to ensure readability? Either we can read an object
or we can't. If we can't, it's _highly_ unlikely that we'll be able to
change permissions on it anyway.

only root can read mode 0 files ... this code exists so that if you're
non-root you can backup and restore mode 0 files which you own... but it
is kind of unfortunate, isn't it?


So I'll just carry this patch on my central server (it's not required on
restore clients), and we can all pretend this didn't happen until the
next person tries to use rdiff-backup to restore from a read-only
location. :-)

i'm tempted to change it so that a command line option is required to
enable the mode futzing... by default i'd really prefer backup and restore
sources be completely read-only.

-dean

Hi,

On Wed, 20 Jul 2005, dean gaudet wrote:

So I'll just carry this patch on my central server (it's not required on
restore clients), and we can all pretend this didn't happen until the
next person tries to use rdiff-backup to restore from a read-only
location. :-)

i'm tempted to change it so that a command line option is required to
enable the mode futzing... by default i'd really prefer backup and restore
sources be completely read-only.

Is it necessary to be able to 'run' off (a copy of) the backup tree, or is
running from a restored tree enough?
I'm using rdiff-backup to backup all files to a normal user account on the
backup server. Only regular files and directories, no symlinks or other
special files. I use find and tar for those 'files', prior to running
rdiff-backup.
The metadata file contains file mode / ownership information, so for me
it's not necessary that all files, directories etc. in the backup tree
have the same permissions as the original ones.

(Note that rdiff-backup is run as root on the server being backed up,
which is the only reasonable thing to do, imho.)


That said, I realize this may not be a useful setup for everyone. If you
require the backup tree to be an exact mirror of the original, having a
metadata file with the correct information doesn't help much when the
trees themselves are different in some respect.

I think having a command line option to choose between a mirror (root
required) and e.g. a backup tree with all files/dirs/... having mode
0600 (files) or 0700 (dirs) (or use umask?) might be nice.


Regards,
Maarten

Well, I think that exactly matters. Try again permitting writing to the log
and report again...

Am Dienstag 19 Juli 2005 17:15 schrieb Sheldon Hearn:
On Monday 18 July 2005 22:49, dean gaudet wrote:
However, the restores exception out on "Operation not permitted",
as per the attached trace.

could you do this with -v5?  i'm always skeptical of the traces that
are reported when remote is involved...

Sure, thanks for taking an interest.

The warning about being unable to write to the log is included for
completeness, but I think it's a non-issue, since it's only a warning
and it shouldn't matter for a restore. It's the "Operation not
permitted" that's getting in my way.

There's nothing special about the packages directory, by the way.
yesterday, it was a different directory. It seems to be the first
directory for which the restore operation would actually do something
to the local target directory.

Ciao,
Sheldon.

--
Vorteile von OpenOffice:
- GRATIS
- einfach zu bedienen
- kompatibel
- portabel
Infos: http://de.openoffice.org/marketing/collaterals/faltflyer_11.pdf

Ich schreibe alle Arbeiten damit.

On Wednesday 20 July 2005 21:30, dean gaudet wrote:
i'm tempted to change it so that a command line option is required to
enable the mode futzing... by default i'd really prefer backup and
restore sources be completely read-only.

Sounds good, although you might want to reverse the logic to maintain
backward compatibility. After all, how many people have come forward
with my complaint? :-)

Ciao,
Sheldon.