Backup Central

Brian Cuttler wrote:
Jean-Louis,

This is the first amdump run with amanda build 3.2.0.svn.3663
on my Solaris 10x86 amanda server.

Which logs can I send to help with the debugging ?

The amdump.1 log file and the first chunker.*.debug that failed, with
the associated dumper.*.debug file.

Jean-Louis

What's the output of:
$amgetconf curie reserved-udp-port
$amgetconf curie reserved-tcp-port
$amgetconf curie unreserved-tcp-port

Jean-Louis

Brian Cuttler wrote:
Jean-Louis,

Looking a little more closely/smartly, the chunker file with
the error is for DLE /export2/samba/sharcor, which seems to
have made it to tape successfully.

I think we recovered from some chunker errors, for DLEs that
entirely failed I'm not finding any chunker*.debug files.


I am attaching

chunker.20101202000224.debug - the oldest chunker file and
one that appears 'normal' to me.

chunker.20101202001735.debug - the oldest chunker file I can
find that looks to have errors.

amdump.1 from last night.

Please let me know if I can send anything else.

amgetconf curie reserved-udp-port
932,948
amgetconf curie reserved-tcp-port
512,1023
amgetconf curie unreserved-tcp-port
10084,10100

Same basic config command I've been using.

We restricted ports when we started placing amanda on FW
systems and to save my sanity I have been building them
with all the same port restrictions for years.

I believe I asked at one point if the port restrictions
limited the server-client threads... I think I was told
'no' but I've always wondered.

more config.log
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by configure, which was
generated by GNU Autoconf 2.63. Invocation command line was

$ ./configure --with-user=amanda --with-group=sys --with-udpportrange=932,948
--with-tcpportrange=10084,10100 --with-gnutar=/usr/sfw/bin/gtar --with-gnuplot=/
opt/sfw/bin/gnuplot --without-libiconv-prefix --without-libintl-prefix LDFLAGS=-
L/usr/sfw/lib -L/opt/sfw/lib -R/usr/sfw/lib -R/opt/sfw/lib CPPFLAGS=-I/usr/sfw/i
nclude -I/opt/sfw/include CFLAGS=-I/usr/sfw/include -I/opt/sfw/include -I/usr/lo
cal/include CC=/opt/SUNWspro/bin/cc EGREP=/usr/sfw/bin/gegrep



On Thu, Dec 02, 2010 at 11:57:13AM -0500, Jean-Louis Martineau wrote:
What's the output of:
$amgetconf curie reserved-udp-port
$amgetconf curie reserved-tcp-port
$amgetconf curie unreserved-tcp-port

Jean-Louis

Brian Cuttler wrote:
Jean-Louis,

Looking a little more closely/smartly, the chunker file with
the error is for DLE /export2/samba/sharcor, which seems to
have made it to tape successfully.

I think we recovered from some chunker errors, for DLEs that
entirely failed I'm not finding any chunker*.debug files.


I am attaching

chunker.20101202000224.debug - the oldest chunker file and
one that appears 'normal' to me.

chunker.20101202001735.debug - the oldest chunker file I can
find that looks to have errors.

amdump.1 from last night.

Please let me know if I can send anything else.


---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

Brian Cuttler wrote:


amgetconf curie reserved-udp-port

932,948

amgetconf curie reserved-tcp-port

512,1023

amgetconf curie unreserved-tcp-port

10084,10100

You have inparallel set to 18 with bsd auth, so the range must be larger
18*5 = 90
Smaller range can also works, but 18 is too small.

You can set it in amanda.conf with unreserved-tcp-port

Jean-Louis

Increase the number of open file for the amanda user.

Jean-Louis

Brian Cuttler wrote:
Jean Louis,

Without recompiling I set the unreserved-tcp-port value to its
default value in amanda.conf on the server.

per (which you contributed to)
http://networking.itags.org/networking-tech/254203/
the range is 1025,65535.

We are back to the errors seen under 3.1.2.

Amanda Backup Client Hosts Check
--------------------------------
ERROR: finsen: service selfcheck: selfcheck: error [open pipe to /usr/local/libexec/amanda/application/amzfs-snapshot: Too many open files]
ERROR: finsen: service /usr/local/libexec/amanda/selfcheck failed: pid 5523 exited with code 1
WARNING: daver-desktop-test: selfcheck request failed: Connection timed out
WARNING: dnixjr: selfcheck request failed: timeout waiting for ACK
Client check: 36 hosts checked in 684.429 seconds. 4 problems found.

(brought to you by Amanda 3.2.0.svn.3663)

However I have made no changes to client 'finsen' which is
still at v3.1.2.

Should I include a client side amanda conf there or is this
an unrelated issue.

from the client side sendsize.20101201183015.debug

Please let me know if you'd like more. I think open-files is
unrelated to the port issue. There is a max-open-files set
someplace ?


Wed Dec 1 18:31:44 2010: sendsize: Spawning "/usr/local/libexec/amanda/applicat
ion/amzfs-snapshot amzfs-snapshot POST-DLE-ESTIMATE --execute-where client --con
fig curie --host finsen --disk /export2/samba/stockdb --device /export2/samba/st
ockdb --level 0 --level 1 --df-path /usr/sbin/df --pfexec YES --pfexec-path /usr
/bin/pfexec --zfs-path /usr/sbin/zfs" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: calculating for amname /export2/samba/watson
, dirname /export2/samba/watson, spindle -1 amgtar
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to /usr/l
ocal/libexec/amanda/application/amzfs-snapshot: Too many open files]
Wed Dec 1 18:31:44 2010: sendsize: Spawning "/usr/local/libexec/amanda/applicat
ion/amgtar amgtar support --config curie --host finsen --disk /export2/samba/wat
son --device /export2/samba/watson" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to /usr/l
ocal/libexec/amanda/application/amgtar: Too many open files]
Wed Dec 1 18:33:06 2010: sendsize: 51811328
Wed Dec 1 18:33:06 2010: sendsize: .....
Wed Dec 1 18:33:06 2010: sendsize: estimate time for / level 1: 151.440
Wed Dec 1 18:33:06 2010: sendsize: estimate size for / level 1: 50597 KB
Wed Dec 1 18:33:06 2010: sendsize: asking killpgrp to terminate
Wed Dec 1 18:33:07 2010: sendsize: done with amname / dirname / spindle -1



thank you,

Brian


On Thu, Dec 02, 2010 at 12:25:25PM -0500, Jean-Louis Martineau wrote:

Brian Cuttler wrote:



amgetconf curie reserved-udp-port


932,948


amgetconf curie reserved-tcp-port


512,1023


amgetconf curie unreserved-tcp-port


10084,10100


You have inparallel set to 18 with bsd auth, so the range must be larger
18*5 = 90
Smaller range can also works, but 18 is too small.

You can set it in amanda.conf with unreserved-tcp-port

Jean-Louis

---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

Jean Louis,

Without recompiling I set the unreserved-tcp-port value to its
default value in amanda.conf on the server.

per (which you contributed to)
http://networking.itags.org/networking-tech/254203/
the range is 1025,65535.

We are back to the errors seen under 3.1.2.

Amanda Backup Client Hosts Check
--------------------------------
ERROR: finsen: service selfcheck: selfcheck: error [open pipe to /usr/local/libexec/amanda/application/amzfs-snapshot: Too many open files]
ERROR: finsen: service /usr/local/libexec/amanda/selfcheck failed: pid 5523 exited with code 1
WARNING: daver-desktop-test: selfcheck request failed: Connection timed out
WARNING: dnixjr: selfcheck request failed: timeout waiting for ACK
Client check: 36 hosts checked in 684.429 seconds. 4 problems found.

(brought to you by Amanda 3.2.0.svn.3663)

However I have made no changes to client 'finsen' which is
still at v3.1.2.

Should I include a client side amanda conf there or is this
an unrelated issue.

from the client side sendsize.20101201183015.debug

Please let me know if you'd like more. I think open-files is
unrelated to the port issue. There is a max-open-files set
someplace ?


Wed Dec 1 18:31:44 2010: sendsize: Spawning "/usr/local/libexec/amanda/applicat
ion/amzfs-snapshot amzfs-snapshot POST-DLE-ESTIMATE --execute-where client --con
fig curie --host finsen --disk /export2/samba/stockdb --device /export2/samba/st
ockdb --level 0 --level 1 --df-path /usr/sbin/df --pfexec YES --pfexec-path /usr
/bin/pfexec --zfs-path /usr/sbin/zfs" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: calculating for amname /export2/samba/watson
, dirname /export2/samba/watson, spindle -1 amgtar
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to /usr/l
ocal/libexec/amanda/application/amzfs-snapshot: Too many open files]
Wed Dec 1 18:31:44 2010: sendsize: Spawning "/usr/local/libexec/amanda/applicat
ion/amgtar amgtar support --config curie --host finsen --disk /export2/samba/wat
son --device /export2/samba/watson" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to /usr/l
ocal/libexec/amanda/application/amgtar: Too many open files]
Wed Dec 1 18:33:06 2010: sendsize: 51811328
Wed Dec 1 18:33:06 2010: sendsize: .....
Wed Dec 1 18:33:06 2010: sendsize: estimate time for / level 1: 151.440
Wed Dec 1 18:33:06 2010: sendsize: estimate size for / level 1: 50597 KB
Wed Dec 1 18:33:06 2010: sendsize: asking killpgrp to terminate
Wed Dec 1 18:33:07 2010: sendsize: done with amname / dirname / spindle -1



thank you,

Brian


On Thu, Dec 02, 2010 at 12:25:25PM -0500, Jean-Louis Martineau wrote:
Brian Cuttler wrote:


amgetconf curie reserved-udp-port

932,948

amgetconf curie reserved-tcp-port

512,1023

amgetconf curie unreserved-tcp-port

10084,10100

You have inparallel set to 18 with bsd auth, so the range must be larger
18*5 = 90
Smaller range can also works, but 18 is too small.

You can set it in amanda.conf with unreserved-tcp-port

Jean-Louis
---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

On Thu, Dec 02, 2010 at 03:14:12PM -0500, Jean-Louis Martineau wrote:
Increase the number of open file for the amanda user.

That is what I thought it would come down to.

setting # limit descriptors in .cshrc seems ineffective
though, since we aren't invoking the shell beneith amandad
on the client.


nor does the following work on solaris...
$ mcr authorize
mod amanda /fillm=1024

Do I have to set the limit system wide ?


Jean-Louis

Brian Cuttler wrote:
Jean Louis,

Without recompiling I set the unreserved-tcp-port value to its
default value in amanda.conf on the server.

per (which you contributed to)
http://networking.itags.org/networking-tech/254203/
the range is 1025,65535.

We are back to the errors seen under 3.1.2.

Amanda Backup Client Hosts Check
--------------------------------
ERROR: finsen: service selfcheck: selfcheck: error [open pipe to
/usr/local/libexec/amanda/application/amzfs-snapshot: Too many open files]
ERROR: finsen: service /usr/local/libexec/amanda/selfcheck failed: pid
5523 exited with code 1
WARNING: daver-desktop-test: selfcheck request failed: Connection timed out
WARNING: dnixjr: selfcheck request failed: timeout waiting for ACK
Client check: 36 hosts checked in 684.429 seconds. 4 problems found.

(brought to you by Amanda 3.2.0.svn.3663)

However I have made no changes to client 'finsen' which is
still at v3.1.2.

Should I include a client side amanda conf there or is this
an unrelated issue.

from the client side sendsize.20101201183015.debug

Please let me know if you'd like more. I think open-files is
unrelated to the port issue. There is a max-open-files set
someplace ?


Wed Dec 1 18:31:44 2010: sendsize: Spawning
"/usr/local/libexec/amanda/applicat
ion/amzfs-snapshot amzfs-snapshot POST-DLE-ESTIMATE --execute-where client
--con
fig curie --host finsen --disk /export2/samba/stockdb --device
/export2/samba/st
ockdb --level 0 --level 1 --df-path /usr/sbin/df --pfexec YES
--pfexec-path /usr
/bin/pfexec --zfs-path /usr/sbin/zfs" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: calculating for amname
/export2/samba/watson
, dirname /export2/samba/watson, spindle -1 amgtar
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to
/usr/l
ocal/libexec/amanda/application/amzfs-snapshot: Too many open files]
Wed Dec 1 18:31:44 2010: sendsize: Spawning
"/usr/local/libexec/amanda/applicat
ion/amgtar amgtar support --config curie --host finsen --disk
/export2/samba/wat
son --device /export2/samba/watson" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to
/usr/l
ocal/libexec/amanda/application/amgtar: Too many open files]
Wed Dec 1 18:33:06 2010: sendsize: 51811328
Wed Dec 1 18:33:06 2010: sendsize: .....
Wed Dec 1 18:33:06 2010: sendsize: estimate time for / level 1: 151.440
Wed Dec 1 18:33:06 2010: sendsize: estimate size for / level 1: 50597 KB
Wed Dec 1 18:33:06 2010: sendsize: asking killpgrp to terminate
Wed Dec 1 18:33:07 2010: sendsize: done with amname / dirname / spindle -1



thank you,

Brian


On Thu, Dec 02, 2010 at 12:25:25PM -0500, Jean-Louis Martineau wrote:

Brian Cuttler wrote:



amgetconf curie reserved-udp-port


932,948


amgetconf curie reserved-tcp-port


512,1023


amgetconf curie unreserved-tcp-port


10084,10100


You have inparallel set to 18 with bsd auth, so the range must be larger
18*5 = 90
Smaller range can also works, but 18 is too small.

You can set it in amanda.conf with unreserved-tcp-port

Jean-Louis

---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.




---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.

Dan,
Jean-Louis,

I realized pretty quickly that using # limit in the .cshrc
would not help much, its not executed by amandad.

Being Solaris I tried to set resource controls and I did
so in the /etc/project file, I added the following line.

The second field, project id number, just happens to match
the amanda socket in /etc/services, its coincidental and
just a project number I thought would be available.

user.amanda:10080::amanda::process.max-file-descriptor=(basic,4096,deny)

Using the command # projects -l, we have the following new output.

user.amanda
projid : 10080
comment: ""
users : amanda
groups : (none)
attribs: process.max-file-descriptor=(basic,4096,deny)

Logging in as the amanda user I find that the output of limit
includes "descriptors 4096, where earlier it had shown 256.

The amanda server continues to report

ERROR: finsen: service selfcheck: selfcheck: error [open pipe to /usr/local/libexec/amanda/application/amzfs-snapshot: Too many open files]
ERROR: finsen: service /usr/local/libexec/amanda/selfcheck failed: pid 21850 exited with code 1

And we have the following in the client's ("finsen") /var/adm/messages

Dec 6 16:33:29 finsen selfcheck[21850]: [ID 702911 daemon.error] error [open pipe to /usr/local/libexec/amanda/application/amzfs-snapshot: Too many open files]

We are _way_ above 256, so I must be missing something.

Does amzfs-snapshot run as root or as amanda ?
Is there something fundamental about inet daemon's that I'm missing ?

thanks,

Brian






On Thu, Dec 02, 2010 at 03:14:12PM -0500, Jean-Louis Martineau wrote:
Increase the number of open file for the amanda user.

Jean-Louis

Brian Cuttler wrote:
Jean Louis,

Without recompiling I set the unreserved-tcp-port value to its
default value in amanda.conf on the server.

per (which you contributed to)
http://networking.itags.org/networking-tech/254203/
the range is 1025,65535.

We are back to the errors seen under 3.1.2.

Amanda Backup Client Hosts Check
--------------------------------
ERROR: finsen: service selfcheck: selfcheck: error [open pipe to
/usr/local/libexec/amanda/application/amzfs-snapshot: Too many open files]
ERROR: finsen: service /usr/local/libexec/amanda/selfcheck failed: pid
5523 exited with code 1
WARNING: daver-desktop-test: selfcheck request failed: Connection timed out
WARNING: dnixjr: selfcheck request failed: timeout waiting for ACK
Client check: 36 hosts checked in 684.429 seconds. 4 problems found.

(brought to you by Amanda 3.2.0.svn.3663)

However I have made no changes to client 'finsen' which is
still at v3.1.2.

Should I include a client side amanda conf there or is this
an unrelated issue.

from the client side sendsize.20101201183015.debug

Please let me know if you'd like more. I think open-files is
unrelated to the port issue. There is a max-open-files set
someplace ?


Wed Dec 1 18:31:44 2010: sendsize: Spawning
"/usr/local/libexec/amanda/applicat
ion/amzfs-snapshot amzfs-snapshot POST-DLE-ESTIMATE --execute-where client
--con
fig curie --host finsen --disk /export2/samba/stockdb --device
/export2/samba/st
ockdb --level 0 --level 1 --df-path /usr/sbin/df --pfexec YES
--pfexec-path /usr
/bin/pfexec --zfs-path /usr/sbin/zfs" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: calculating for amname
/export2/samba/watson
, dirname /export2/samba/watson, spindle -1 amgtar
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to
/usr/l
ocal/libexec/amanda/application/amzfs-snapshot: Too many open files]
Wed Dec 1 18:31:44 2010: sendsize: Spawning
"/usr/local/libexec/amanda/applicat
ion/amgtar amgtar support --config curie --host finsen --disk
/export2/samba/wat
son --device /export2/samba/watson" in pipeline
Wed Dec 1 18:31:44 2010: sendsize: critical (fatal): error [open pipe to
/usr/l
ocal/libexec/amanda/application/amgtar: Too many open files]
Wed Dec 1 18:33:06 2010: sendsize: 51811328
Wed Dec 1 18:33:06 2010: sendsize: .....
Wed Dec 1 18:33:06 2010: sendsize: estimate time for / level 1: 151.440
Wed Dec 1 18:33:06 2010: sendsize: estimate size for / level 1: 50597 KB
Wed Dec 1 18:33:06 2010: sendsize: asking killpgrp to terminate
Wed Dec 1 18:33:07 2010: sendsize: done with amname / dirname / spindle -1



thank you,

Brian


On Thu, Dec 02, 2010 at 12:25:25PM -0500, Jean-Louis Martineau wrote:

Brian Cuttler wrote:



amgetconf curie reserved-udp-port


932,948


amgetconf curie reserved-tcp-port


512,1023


amgetconf curie unreserved-tcp-port


10084,10100


You have inparallel set to 18 with bsd auth, so the range must be larger
18*5 = 90
Smaller range can also works, but 18 is too small.

You can set it in amanda.conf with unreserved-tcp-port

Jean-Louis

---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.




---
Brian R Cuttler brian.cuttler < at > wadsworth.org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.