Since December 22, 2011 many of my backup clients have had problems with the Networker save.exe and savepnpc.exe processes crashing at the start of a backup. Their Windows application event logs have related entries stating "Faulting application save.exe, version 126.96.36.199, faulting module ntdll.dll, version 5.2.3790.4937, fault address 0x0002bcae." The client version varies as stated below. The address varies. The faulting module is either msvcr80.dll or ntdll.dll.
The same client is not affected every backup. Increasing the job retry count gets around the problem as the subsequent attempts might succeed.
This has happened on 25 out of 62 clients. These clients are all Windows. Most are Windows Server 2003 SP2, other are Windows Server 2008 R2. This affects Windows 2003 clients more often. All clients receive Microsoft patches every 1 to 2 months. There were no patches applied for several weeks prior to the first event. We intentionally did not make changes during December due to staff vacations.
The Networker client versions range from 7.4.4 to 188.8.131.52. Every version of the client has had the problem.
Antivirus software is Trend OfficeScan 10.5.1083. There were no antivirus application updates during the time the problem started. Virus definitions are pushed out immediately after they're received from Trend. The antivirus client was updated to 10.5.2122 last week on several clients, and the problem persists.
I have a case open with EMC. They have my crash dumps, the above history, and more details. Unfortunately, they haven't found the problem. They suggested that the heap size was too small, and recommended the changes found in Microsoft kb126962. This was applied to 3 clients which had the same problem within a few days. EMC recommended upgrading the clients to 184.108.40.206. This was applied to 3 clients to no avail.
For any help, thank you very much.