Backup Central

Since December 22, 2011 many of my backup clients have had problems with the Networker save.exe and savepnpc.exe processes crashing at the start of a backup. Their Windows application event logs have related entries stating "Faulting application save.exe, version 7.6.1.0, faulting module ntdll.dll, version 5.2.3790.4937, fault address 0x0002bcae." The client version varies as stated below. The address varies. The faulting module is either msvcr80.dll or ntdll.dll.

The same client is not affected every backup. Increasing the job retry count gets around the problem as the subsequent attempts might succeed.

This has happened on 25 out of 62 clients. These clients are all Windows. Most are Windows Server 2003 SP2, other are Windows Server 2008 R2. This affects Windows 2003 clients more often. All clients receive Microsoft patches every 1 to 2 months. There were no patches applied for several weeks prior to the first event. We intentionally did not make changes during December due to staff vacations.

The Networker client versions range from 7.4.4 to 7.6.3.3. Every version of the client has had the problem.

Antivirus software is Trend OfficeScan 10.5.1083. There were no antivirus application updates during the time the problem started. Virus definitions are pushed out immediately after they're received from Trend. The antivirus client was updated to 10.5.2122 last week on several clients, and the problem persists.

I have a case open with EMC. They have my crash dumps, the above history, and more details. Unfortunately, they haven't found the problem. They suggested that the heap size was too small, and recommended the changes found in Microsoft kb126962. This was applied to 3 clients which had the same problem within a few days. EMC recommended upgrading the clients to 7.6.3.3. This was applied to 3 clients to no avail.

Any ideas?

For any help, thank you very much.

Nic Romero

Hi Nic,

unfortunately, you did not state the NW server version. I hope that it is not higher than the client' one. If so, EMC should have pointed out that this is not supported.

Besides that, let me recommend that you start the backup only when the AV scan has finished.
You should also disable it for the NW server.

If a higher retry count improves the behavior, this usually indicates a 'network related problem'. As this 'area' is huge, i cannot be more precise.
However, retries usually appear at the beginning of a backup (in the 'probe' phase), not during the save process itself. During the save process '(inactivity) timeout problems' are more likely to occur. To test the exact origin let me recommend that you use larger manual backups ('bigasm' would be ideal) to avoid any other processes. Using the command line also allows to display a more verbose output.

> unfortunately, you did not state the NW server version.
> I hope that it is not higher than the client' one. If so, EMC
> should have pointed out that this is not supported.

The server version is 7.6.1. I have clients older than the server, but they crash no more often than the later clients. EMC hasn't said anything about upgrading the older clients. I was in the process of upgrading all the clients to the latest version, but when that didn't solve the problem I stopped. I figured I might as well wait until I get a stable version.

> Besides that, let me recommend that you start
> the backup only when the AV scan has finished.

The scheduled AV scan doesn't run during backups. It runs once a week, while the app crashes happen throughout the week. Real time scan, on the other hand, is always on. EMC did recommend once disabling AV, but I hesitate to open up vulnerabilities for an unspecified when the problem occurs randomly. However, I will see about disabling real time scan during backups on the more chronic clients.

> If a higher retry count improves the behavior, this usually indicates a
> 'network related problem'.

Well, I wouldn't go so far as to say the situation is improved. The client still crashes, but the server attempts another backup of the same saveset. The more often I try, the better chances I'll get an attempt in which the client does not crash.

Here are lines from daemon.raw relating to one client's savepnpc.exe crashing. I clipped only the parts that seems pertinent.
- otdmsas01.otlan1.treasurer.ca.gov:D:\ unexpectedly exited.

- otdmsas01.otlan1.treasurer.ca.gov:D:\ Cannot determine status of backup process. Use mminfo to determine job status.

- savegrp command 'savepnpc -s otbrs01.otlan1.treasurer.ca.gov -g "Oracle Servers - Begin" -LL -f - -m otdmsas01.otlan1.treasurer.ca.gov -t 1336811523 -l incr -q -W 78 -N "D:\\" "D:\\"' for client otdmsas01.otlan1.treasurer.ca.gov exited with return code 11

These errors look more like a problem DURING the backup itself (save).

Once again, let me recommend:
- no client should have a higher version number than the server. It could simply be that the client uses a 'feature' which the older version of the server/SN can not offer yet.
- Try to isolate the problem. Test with manual backups first. If they will not run, do not expect server-initiated backups to run.

> - no client should have a higher version number than the server.
Let me run that by EMC support staff since they're the ones recommending I upgrade the clients to the latest revision, without mention of the server revision. Worst-case scenario is I upgrade the server. Do you think this is something I should do before continuing other troubleshooting steps?

> - Try to isolate the problem. Test with manual backups first.
I've done manual backups on some of the affected clients. They worked fine. Unfortunately, due to the intermittent nature of the problem, I didn't get any useful information. However, what I will do is take the worst client, and setup something where all I run are manual backups for a week, with ultimate verbosity enabled.

Thank you for your recommendations.

As you have updated the clients already, it does not make sense to go back. If you are lucky, the new server version might already solve the problem.