Backup Central

While trying to implement the previously-discussed feature (adding a
prefix to all remotely-provided paths in server mode), I've come upon
what appears to be a bug in the server-mode security model.

Tue Aug 16 00:24:51 2005 Server sending (0): None
Tue Aug 16 00:24:51 2005 Client received (0): None
Tue Aug 16 00:24:51 2005 Making directory backup-files
Tue Aug 16 00:24:51 2005 Client sending (0): ConnectionRequest:
os.mkdir with 1 arguments
Tue Aug 16 00:24:51 2005 Client sending (0): 'backup-files'
Tue Aug 16 00:24:51 2005 Server received (0): ConnectionRequest:
os.mkdir with 1 arguments
Tue Aug 16 00:24:51 2005 Server received (0): 'backup-files'
Vetting request (ConnectionRequest: os.mkdir with 1 arguments),
['backup-files']
Not vetting backup-files against restricted path list

The last few lines are from my own instrumentation -- but nonetheless,
it appears quite clearly that 'backup-files' is in this case passed as
type str rather than as an RPath object, and thus that it never makes it
to Security.vet_rpath(). This appears to be the case with other requests
as well -- so far os.mkdir, os.listdir, os.chmod and C.make_file_dict,
though I do not pretend to know of the exhaustiveness of the above list.
This strikes me as a rather Bad Thing.

---------------

Going back to the path-prefixing feature, I have a patch that appears to
work under *very light* testing, but it's just intended as a
proof-of-concept -- I have very little confidence in its correctness.
Ben, I would very much appreciate any input you could provide. I would
like to get this deployed to my QA department quickly, and will gladly
spend time writing code as needed to do so -- but some guidance and
advice would be exceedingly helpful.

Thank you kindly!

diff -ru rdiff-backup-1.0.0/rdiff_backup/Globals.py rdiff-backup-1.0.0/rdiff_backup.new/Globals.py
--- rdiff-backup-1.0.0/rdiff_backup/Globals.py 2005-08-14 01:12:55.000000000 -0500
+++ rdiff-backup-1.0.0/rdiff_backup.new/Globals.py 2005-08-15 11:55:14.000000000 -0500
< at > < at > -215,6 +215,9 < at > < at >
# If set, exit with error instead of dropping ACLs or ACL entries.
never_drop_acls = None

+# If running as a server, append this prefix to all paths used.
+path_prefix = ""
+

def get(name):
"""Return the value of something in this module"""
diff -ru rdiff-backup-1.0.0/rdiff_backup/Main.py rdiff-backup-1.0.0/rdiff_backup.new/Main.py
--- rdiff-backup-1.0.0/rdiff_backup/Main.py 2005-08-14 01:12:55.000000000 -0500
+++ rdiff-backup-1.0.0/rdiff_backup.new/Main.py 2005-08-15 23:55:12.000000000 -0500
< at > < at > -62,7 +62,7 < at > < at >
"exclude-filelist-stdin", "exclude-globbing-filelist=",
"exclude-globbing-filelist-stdin", "exclude-mirror=",
"exclude-other-filesystems", "exclude-regexp=",
- "exclude-special-files", "force", "group-mapping-file=",
+ "exclude-special-files", "force", "force-path-prefix=", "group-mapping-file=",
"include=", "include-filelist=", "include-filelist-stdin",
"include-globbing-filelist=",
"include-globbing-filelist-stdin", "include-regexp=",
< at > < at > -115,6 +115,7 < at > < at >
"standard input"))
select_files.append(sys.stdin)
elif opt == "--force": force = 1
+ elif opt == "--force-path-prefix": Globals.path_prefix = normalize_path(arg)
elif opt == "--group-mapping-file": group_mapping_filename = arg
elif (opt == "--include" or
opt == "--include-special-files" or
diff -ru rdiff-backup-1.0.0/rdiff_backup/Security.py rdiff-backup-1.0.0/rdiff_backup.new/Security.py
--- rdiff-backup-1.0.0/rdiff_backup/Security.py 2005-08-14 01:12:55.000000000 -0500
+++ rdiff-backup-1.0.0/rdiff_backup.new/Security.py 2005-08-16 00:55:24.426435000 -0500
< at > < at > -21,6 +21,7 < at > < at >

import sys, tempfile
import Globals, Main, rpath, log
+import os.path

class Violation(Exception):
"""Exception that indicates an improper request has been received"""
< at > < at > -177,11 +178,20 < at > < at >

def vet_request(request, arglist):
"""Examine request for security violations"""
- #if Globals.server: sys.stderr.write(str(request) + "\n")
+ #if Globals.server: sys.stderr.write("Vetting request (%s), %s [%s]\n" % (str(request), str(arglist), repr([type(arg) for arg in arglist])))
security_level = Globals.security_level
+ if Globals.path_prefix:
+ for arg in arglist:
+ if isinstance(arg, rpath.RPath):
+ #if Globals.server: sys.stderr.write("Adding prefix to RPath (%s,%s)\n" % (repr(arg.base), repr(arg.path)))
+ arg.base = os.path.join(Globals.path_prefix, arg.base)
+ arg.path = os.path.join(Globals.path_prefix, arg.path)
+ elif isinstance(arg, str) and request.function_string in ['os.mkdir', 'os.listdir', 'os.chmod', 'C.make_file_dict']:
+ arglist[arglist.index(arg)] = os.path.join(Globals.path_prefix, arg)
if Globals.restrict_path:
for arg in arglist:
if isinstance(arg, rpath.RPath): vet_rpath(arg)
+ #elif isinstance(arg, str): sys.stderr.write("Not vetting %s against restricted path list\n" % arg)
if security_level == "all": return
if request.function_string in allowed_requests: return
if request.function_string in ("Globals.set", "Globals.set_local"):

Charles Duffy <cduffy < at > spamcop.net>
wrote the following on Tue, 16 Aug 2005 01:07:32 -0500
While trying to implement the previously-discussed feature (adding a
prefix to all remotely-provided paths in server mode), I've come upon
what appears to be a bug in the server-mode security model.

Tue Aug 16 00:24:51 2005 Server sending (0): None
Tue Aug 16 00:24:51 2005 Client received (0): None
Tue Aug 16 00:24:51 2005 Making directory backup-files
Tue Aug 16 00:24:51 2005 Client sending (0): ConnectionRequest:
os.mkdir with 1 arguments
Tue Aug 16 00:24:51 2005 Client sending (0): 'backup-files'
Tue Aug 16 00:24:51 2005 Server received (0): ConnectionRequest:
os.mkdir with 1 arguments
Tue Aug 16 00:24:51 2005 Server received (0): 'backup-files'
Vetting request (ConnectionRequest: os.mkdir with 1 arguments),
['backup-files']
Not vetting backup-files against restricted path list

The last few lines are from my own instrumentation -- but nonetheless,
it appears quite clearly that 'backup-files' is in this case passed as
type str rather than as an RPath object, and thus that it never makes it
to Security.vet_rpath(). This appears to be the case with other requests
as well -- so far os.mkdir, os.listdir, os.chmod and C.make_file_dict,
though I do not pretend to know of the exhaustiveness of the above list.
This strikes me as a rather Bad Thing.

But C.make_file_dict, os.mkdir, etc. aren't (or at least shouldn't be)
in the list of allowable commands. When a security level is active,
the first check is to make sure the command is allowed. If it is,
then the rpaths are checked to see if they start with the right
prefix.

Going back to the path-prefixing feature, I have a patch that appears to
work under *very light* testing, but it's just intended as a
proof-of-concept -- I have very little confidence in its correctness.
Ben, I would very much appreciate any input you could provide. I would
like to get this deployed to my QA department quickly, and will gladly
spend time writing code as needed to do so -- but some guidance and
advice would be exceedingly helpful.

Personally I don't experience managing user-run rdiff-backup sessions
in a secure environment so I didn't reply to your earlier message.
(Others on the list would know more about the general setup.) But I
didn't understand why you wanted to patch rdiff-backup instead of
running a more standard setup, like that described on Dean Gaudet's
page at:

http://arctic.org/~dean/rdiff-backup/unattended.html

So instead of restricting by path, it seems safer and easier to
restrict by ssh-key and/or username.


--
Ben Escoto

On Tue, 2005-08-16 at 23:24 -0500, Ben Escoto wrote:
Charles Duffy <cduffy < at > spamcop.net>
wrote the following on Tue, 16 Aug 2005 01:07:32 -0500
While trying to implement the previously-discussed feature (adding a
prefix to all remotely-provided paths in server mode), I've come upon
what appears to be a bug in the server-mode security model.

Tue Aug 16 00:24:51 2005 Server sending (0): None
Tue Aug 16 00:24:51 2005 Client received (0): None
Tue Aug 16 00:24:51 2005 Making directory backup-files
Tue Aug 16 00:24:51 2005 Client sending (0): ConnectionRequest:
os.mkdir with 1 arguments
Tue Aug 16 00:24:51 2005 Client sending (0): 'backup-files'
Tue Aug 16 00:24:51 2005 Server received (0): ConnectionRequest:
os.mkdir with 1 arguments
Tue Aug 16 00:24:51 2005 Server received (0): 'backup-files'
Vetting request (ConnectionRequest: os.mkdir with 1 arguments),
['backup-files']
Not vetting backup-files against restricted path list

The last few lines are from my own instrumentation -- but nonetheless,
it appears quite clearly that 'backup-files' is in this case passed as
type str rather than as an RPath object, and thus that it never makes it
to Security.vet_rpath(). This appears to be the case with other requests
as well -- so far os.mkdir, os.listdir, os.chmod and C.make_file_dict,
though I do not pretend to know of the exhaustiveness of the above list.
This strikes me as a rather Bad Thing.

But C.make_file_dict, os.mkdir, etc. aren't (or at least shouldn't be)
in the list of allowable commands. When a security level is active,
the first check is to make sure the command is allowed. If it is,
then the rpaths are checked to see if they start with the right
prefix.

If they aren't in the list of allowable commands, why am I seeing the
client sending such requests and the server processing them? I don't
thoroughly understand at what times and under what circumstances
security levels are active, but (without better understanding what's
going on) the behaviour in question seems a touch suspect.

Going back to the path-prefixing feature, I have a patch that appears to
work under *very light* testing, but it's just intended as a
proof-of-concept -- I have very little confidence in its correctness.
Ben, I would very much appreciate any input you could provide. I would
like to get this deployed to my QA department quickly, and will gladly
spend time writing code as needed to do so -- but some guidance and
advice would be exceedingly helpful.

Personally I don't experience managing user-run rdiff-backup sessions
in a secure environment so I didn't reply to your earlier message.
(Others on the list would know more about the general setup.) But I
didn't understand why you wanted to patch rdiff-backup instead of
running a more standard setup, like that described on Dean Gaudet's
page at:

http://arctic.org/~dean/rdiff-backup/unattended.html

So instead of restricting by path, it seems safer and easier to
restrict by ssh-key and/or username.

Not workable in my situation:

- The instructions from the page in question require work to be done on
a per-server basis. I need to support tens to hundreds (and possibly
someday thousands) of servers with minimal administrative overhead.

- I don't trust the servers to accurately identify themselves (ie. to
choose locations under the backup account on which to store data). These
servers are in the posession of various clients, and store data
proprietary to said clients. If a client could subvert the backup system
to download another client's data (as is possible when all servers share
a single backup account without per-system pathname limitations), it
would be a Very Bad Thing. Because of the number of servers in question,
creating multiple backup accounts (to isolate the servers from each
other) is likewise unworkable.

- Using SSH as a transport is undesirable. I already have a VPN
providing encrypted, authenticated transport; adding a second layer of
encryption and authentication (or even just tunneling) is redundant,
unnecessarily complicating, and inefficient.

Charles Duffy <cduffy < at > spamcop.net>
wrote the following on Tue, 16 Aug 2005 23:58:22 -0500

If they aren't in the list of allowable commands, why am I seeing the
client sending such requests and the server processing them? I don't
thoroughly understand at what times and under what circumstances
security levels are active, but (without better understanding what's
going on) the behaviour in question seems a touch suspect.

Well unless you use an option like --restrict*, security is pretty
limited: it just tries to prevent a situation where you run
"rdiff-backup source host::dest" and the remote rdiff-backup on host
is actually hacked and tries to read/delete inappropriate files on the
local side. On the destination side everything is fair game.

Anyway, how are you running rdiff-backup? I'll check out the
mkdir()'s and similar you are finding.

- I don't trust the servers to accurately identify themselves (ie. to
choose locations under the backup account on which to store data). These
servers are in the posession of various clients, and store data
proprietary to said clients. If a client could subvert the backup system
to download another client's data (as is possible when all servers share
a single backup account without per-system pathname limitations), it
would be a Very Bad Thing. Because of the number of servers in question,
creating multiple backup accounts (to isolate the servers from each
other) is likewise unworkable.

What's the problem with having thousands of users? It seems that
would be the safest way. Otherwise, why not write a script that
checks the arguments to rdiff-backup, instead of patching
rdiff-backup?


--
Ben Escoto

On Tue, 16 Aug 2005, Charles Duffy wrote:

http://arctic.org/~dean/rdiff-backup/unattended.html

Not workable in my situation:

- The instructions from the page in question require work to be done on
a per-server basis. I need to support tens to hundreds (and possibly
someday thousands) of servers with minimal administrative overhead.

that's my bad really -- you actually only need a single special backup
key... and you can clone the /root/.ssh/authorized_keys2 file everywhere
with that one key.

but you couldn't as easily use the .ssh/config hack i'm using there for
specifying the identityfile for hundreds of hosts... what you'd want to do
is something like this for each "fishie":

rdiff-backup --remote-schema 'ssh -C -i .ssh/backup_key %s rdiff-backup --server' fishie::/ /backup/fishie

i'm not sure why i went with a different key per server...

heck you don't even need the special key, the default key for the
backup user could be used... it's just that if you did this then you'd
have extra hassle when you want to restore.

restoring is another detail my page doesn't go into... for that it's great
to have a key dedicated for restore purposes -- but i really don't like
having those in my .ssh/authorized_keys2 files by default... i prefer
to add them manually when i'm actually doing a restore.

i guess i should revamp the directions sometime :)

-dean

On Tue, 2005-08-16 at 23:30 -0700, dean gaudet wrote:
that's my bad really -- you actually only need a single special backup
key... and you can clone the /root/.ssh/authorized_keys2 file everywhere
with that one key.

Yes, and I realize that -- but it doesn't address the [important] "hosts
shouldn't be able to pull each others' backups" issue, or the [less
important] "SSH is redundant w/ the VPN" issue.

I'm not saying that your solution isn't appropriate for many or most
environments -- but it doesn't seem a good fit for mine.

On Wed, 2005-08-17 at 01:33 -0500, Ben Escoto wrote:
Charles Duffy <cduffy < at > spamcop.net>
wrote the following on Tue, 16 Aug 2005 23:58:22 -0500

If they aren't in the list of allowable commands, why am I seeing the
client sending such requests and the server processing them? I don't
thoroughly understand at what times and under what circumstances
security levels are active, but (without better understanding what's
going on) the behaviour in question seems a touch suspect.

Well unless you use an option like --restrict*, security is pretty
limited: it just tries to prevent a situation where you run
"rdiff-backup source host::dest" and the remote rdiff-backup on host
is actually hacked and tries to read/delete inappropriate files on the
local side. On the destination side everything is fair game.

Hmm. See, my concern is protecting inappropriate files on host from
being accessed. Perhaps I'll want to use a different security layer in
addition to application-based measures.

Anyway, how are you running rdiff-backup? I'll check out the
mkdir()'s and similar you are finding.

On the server:
rdiff-backup --server --restrict "$DATAPATH" --force-path-prefix "$DATAPATH"
(where DATAPATH is based on, among other things, the connecting system's
hostname; this is all behind tcpsvd, http://smarden.org/ipsvd/)

On the client:
rdiff-backup --remote-schema 'netcat %s 10873' /RecareEE/data/backed-up 10.1.128.1::/backup

- I don't trust the servers to accurately identify themselves (ie. to
choose locations under the backup account on which to store data). These
servers are in the posession of various clients, and store data
proprietary to said clients. If a client could subvert the backup system
to download another client's data (as is possible when all servers share
a single backup account without per-system pathname limitations), it
would be a Very Bad Thing. Because of the number of servers in question,
creating multiple backup accounts (to isolate the servers from each
other) is likewise unworkable.

What's the problem with having thousands of users? It seems that
would be the safest way. Otherwise, why not write a script that
checks the arguments to rdiff-backup, instead of patching
rdiff-backup?

- Checking the arguments to rdiff-backup:
Would you do this checking on the server or the client? Remember, I
don't trust the client (which may have been subverted to try to retrieve
another machine's backups), so it needs to be done on the server. Does
the server have a nonspoofable way to check the client's command-line
arguments without patching?

- Passing arguments specific to the client:
One option to workaround this is, on the server, to detect the host
the client connected from and generate a custom restrict path, such as
"--restrict /path/to/backups/for/CLIENTNAME". This means that each
client must be run as something along the lines of
"rdiff-backup /path/to/backed-up-data
server::/path/to/backups/for/MYHOSTNAME". This means that
"/path/to/backups/for" must be agreed upon between client and server,
which is ugly: Why should the client need to know anything about the
server's filesystem structure? And since the host knows who it thinks
the client is, why should the client need to pass in its name as well?

- Extra overhead/complexity from SSH
I brought this concern up earlier, and it hasn't been answered. Why
bother with host keys, RSA authentication, and (unless disabled)
encryption overhead when there's already a VPN in place providing a
guarantee that the hosts are who they say they are?

On Wed, 2005-08-17 at 08:03 +0100, Keith Edmunds wrote:
Charles Duffy wrote:
Hmm. See, my concern is protecting inappropriate files on host from
being accessed. Perhaps I'll want to use a different security layer in
addition to application-based measures.

Charles, I know that you are against each server having its own username
on the backup server, but that is _exactly_ what having individual
server accounts will give you. That's how I implement backups, albeit
for fewer servers than you have, and it works very well. You could spend
a lot of time trying to find the ideal solution...

Keith,

Your concern is appreciated, but I *have* a solution that seems to work
well for me, without the overhead of (potentially) thousands of accounts
and with the additional benefit of eliminating SSH overhead (which is
redundant w/ the VPN I run). The only issue I have is that it requires
that the server have a patched copy of rdiff-backup. I consider this a
reasonable requirement -- but would like to see my patch vetted and
pushed upstream rather than remaining a local modification only.

With regard to additional, lower-level measures, an LD_PRELOAD that vets
file accesses to make sure they're happening within approved paths is
still less administrative overhead than would be involved in having
remote-server:local-account parity.

Charles Duffy wrote:
Hmm. See, my concern is protecting inappropriate files on host from
being accessed. Perhaps I'll want to use a different security layer in
addition to application-based measures.

Charles, I know that you are against each server having its own username
on the backup server, but that is _exactly_ what having individual
server accounts will give you. That's how I implement backups, albeit
for fewer servers than you have, and it works very well. You could spend
a lot of time trying to find the ideal solution...

Keith

--
Keith Edmunds

+---------------------------------------------------------------------+
| Tiger Computing Ltd | Helping businesses make the most of Linux |
| "The Linux Company" | http://www.TheLinuxConsultancy.co.uk |
+---------------------------------------------------------------------+

Charles Duffy <cduffy < at > spamcop.net>
wrote the following on Wed, 17 Aug 2005 01:54:18 -0500

On the server:
rdiff-backup --server --restrict "$DATAPATH" --force-path-prefix "$DATAPATH"
...

Thanks for the report, I can see the bug too (I'm not sure the mkdir
got through, since it isn't on the list, but functions like
C.make_file_dict and os.listdir get through even if they're not in the
--restrict dir). I listed it as bug 14209 on savannah.

What's the problem with having thousands of users? It seems that
would be the safest way. Otherwise, why not write a script that
checks the arguments to rdiff-backup, instead of patching
rdiff-backup?

- Checking the arguments to rdiff-backup:
Would you do this checking on the server or the client? Remember, I
don't trust the client (which may have been subverted to try to retrieve
another machine's backups), so it needs to be done on the server. Does
the server have a nonspoofable way to check the client's command-line
arguments without patching?

Well I don't really understand your setup yet, but it seems any way
you do it, the client will have to authenticate itself to the server
somehow. The script can just check these credentials in the same way
your patched rdiff-backup would.

- Extra overhead/complexity from SSH
I brought this concern up earlier, and it hasn't been answered. Why
bother with host keys, RSA authentication, and (unless disabled)
encryption overhead when there's already a VPN in place providing a
guarantee that the hosts are who they say they are?

You need some way of setting up a pipe from one server to the other.
You could use ssh without encryption, or rsh, or whatever way you were
going to use before (netcat?). Just make sure rdiff-backup gets run
with the right uid on the server.


--
Ben Escoto

Ben Escoto wrote:

Well I don't really understand your setup yet, but it seems any way
you do it, the client will have to authenticate itself to the server
somehow. The script can just check these credentials in the same way
your patched rdiff-backup would.


It's not a matter of credentials -- I trust the client's credentials, or
it wouldn't have been able to make the connection (I *do* have a quite
strong authentication mechanism in place -- that's the whole VPN thing
I've been prattling about). The issue is not allowing a "trusted" client
in posession of a third party (this client being legitimately
authenticated to make and retrieve its own backups) to retrieve files
belonging to another third party. This could easily happen if one of our
client boxen is subverted by the third party who has physical control of
the hardware it runs on.

Consequently, the client should have the ability to connect to the
server and make backup and restore requests -- but those requests should
be applied within its own private namespace. Applying a host-specific
prefix to the path it retrieves files from or stores them to seems the
easiest way to enforce this private-namespace policy; thus, a patched
version of rdiff-backup.

- Extra overhead/complexity from SSH
I brought this concern up earlier, and it hasn't been answered. Why
bother with host keys, RSA authentication, and (unless disabled)
encryption overhead when there's already a VPN in place providing a
guarantee that the hosts are who they say they are?



You need some way of setting up a pipe from one server to the other.


Yes, I have one (netcat on the client, tcpsvd on the server). That's
fine, it works -- but it rules out the (SSH-based) multi-server
authentication mechansims which have been thus far suggested (and which
I don't have a need for anyhow, on account of the VPN).

Charles Duffy <cduffy < at > spamcop.net>
wrote the following on Thu, 18 Aug 2005 06:39:44 -0500

Yes, I have one (netcat on the client, tcpsvd on the server). That's
fine, it works -- but it rules out the (SSH-based) multi-server
authentication mechansims which have been thus far suggested (and which
I don't have a need for anyhow, on account of the VPN).

Well I was just suggesting using usernames.. Like instead of having
the server run

rdiff-backup --restrict XXXX --force-path-prefix XXXX --server

you could have it run

cd XXXX; sudo -u YYYYY rdiff-backup --restrict . --server

to avoid patching rdiff-backup and for an additional layer of
security. Anyway it's up to you, I'm mainly posting to say I think
the security bug is fixed in CVS. Thank you for the report.


--
Ben Escoto

Ben Escoto wrote:

Well I was just suggesting using usernames.. Like instead of having
the server run

rdiff-backup --restrict XXXX --force-path-prefix XXXX --server

you could have it run

cd XXXX; sudo -u YYYYY rdiff-backup --restrict . --server

to avoid patching rdiff-backup and for an additional layer of
security.

Hmm -- that might actually be useful. So I take it the client can use
relative paths (ie. "server::backup" instead of "server::/backup", or at
worst "server::./backup" and it'll be evaluated relative to the cwd)?

Separate per-client user accounts is impractical for my purposes for
reasons I've already discussed -- but using relative paths could indeed
make my patch unnecessary. Thanks for the suggestion!