RPC Error Troubleshooting on Cyberstation

[Product_Support]

Issue

There are a number of reasons that RPC error occur on Cyberstation and troubleshooting steps are needed

Product Line

Andover Continuum

Environment

• Continuum
• Webclient
• Windows XP
• Window Vista
• Windows 7
• Windows Server 2003
• Windows Server 2008

Cause

RPC (Remote Procedure Call) errors happen on Cyberstation when the connection to the SQL database is not available

Resolution

There are many reasons that RPC errors can occur. A Google search of RPC errors will indicate that many applications have similar issues with these errors.

An RPC error on a Cyberstation means that it has lost connection to the database for some reason and you will need to restart Cyberstation in order to recover. Below is a list of things to check when dealing with RPC errors on CyberStation.

1. Multiple Cybers

   Does the issue occur on multiple Cyberstations?

2. Database

   Does the issue follow the database? i.e. load the database onto a test server or try a test blank database. Were some new objects recently added to the database when this started happening?

   Does reverting to a backup stop the errors?

3. Increase Virtual Memory

   Right Click on My Computer> Properties> Advanced System Settings> Under Performance select the Settings button> Advanced tab> Under Virtual Memory click the Change button

   Recommended is 4096

4. Network disruption or slowness
   1. Networking  -  Verify all aspects of your networked environment.  This includes, but not limited to, slow networks, possible problems with your network interface card (NIC), cabling issues, switch/router issues, busy networks.  As you can see from the above definition of what RPC is it can be heavily reliant on the stability of the network and all it’s interfaces.

      Continuum does not support DHCP

   2. Multiple NIC cards – ensure there is only 1 NIC card enabled.

      No all NIC cards are created equal. Connections issue can be caused by an integrated NIC card of low quality. There can be settings within the NIC card itself that could affect consistent communication. Check the manufacturers website for details.

      Verify that there is no duplication of IP addresses on the network. Pull a cable and ping the address, if you still get a response there are 2 devices with the same IP address.

   3. WINS -  Verify your WINS settings and make sure there are no issues with the WINS Server.  This is known by Microsoft to cause RPC errors.

   4. DNS - Verify your DNS settings and make surer there are no issues with the DNS server.  This also is known by Microsoft to cause RPC errors.

   5. Verify that  the "TCP/IP NetBIOS Helper" service is running.

   6. Check and see if “File and Print Sharing” is enabled.  There are references in a lot of online articles, on RPC, problems that turning on “File and Print sharing” resolves RPC errors.

   7. Ping and SQL

      Is the SQL database still available to Cyberstation? From a DOS prompt>

      Ping IP (SQL Server) –t (continuous)

      Ex. c:\> ping 192.168.1.5 –t

      Note the number of responses as well as the time in milliseconds to respond.

      A local SQL server would usually be under 100ms.

      Check resources on the SQL Server (CPU, I/O Read/Write, memory usage, etc.)

   8. Windows Server 2003 Network card setting

      (This setting isn’t available in Windows Server 2008)

      Shut Continuum down on the machine having the problem.
      Start>Settings>Network Connections>Local Area Connections>Right Click and select Properties> Click on File and Printer Sharing for Microsoft Networks>Click Properties button> Select 'Maximize data throughput for network applications'.

      (Default is Maximize data throughput for File Sharing)
      Ok through all the screens and start Continuum and see if this makes a difference.

      
      

      XP/Vista/Windows 7

      Start> Settings> Network Connections> Local Area Connection> Properties>

      Advanced tab> Windows Firewall - Click the Setting button> Click the

      exceptions tab

      Verify that File and Print Sharing is selected.

      File and printer sharing are disabled by default with XP SP2.

      There is no network throughput option in XP, Vista or Windows 7 like there is on Windows Server 2003.

5. Windows Event Viewer

   Take a look at the Windows Event Viewer Application Log (Start> Run type ‘eventvwr’ or Launch Windows Control Panel> Administrative Tools> Event Viewer

   Examine the events and look for RPC error indications in the details. There may be a pattern of which executable is causing the errors.

   You may also see informational events that indicated momentary disconnects. A large number of these can indicate an issue with the Cyberstation being able to connect to the database consistently in a timely manner.

6. Monitoring tools

   Run an acctrace.exe (C:\Program Files\Continuum) and a wireshark capture at the time of the RPC error to see the Continuum and network processing going on at the Cyberstation.

   Windows Process Monitor may also be helpful in seeing which process is having trouble.

   http://technet.microsoft.com/en-us/sysinternals/bb896645

7. Check the system for SpyWare and/or Viruses.
8. Workgroup versus Domain connection

   An excerpt from the Continuum System Admin manual

   In a LAN Configuration, the Continuum CyberStation must be in a “Microsoft Domain “

   type installation. This is because the Continuum CyberStation will use Named Pipes

   Interprocess Communication System (IPC). This is accomplished with Windows remote

   procedure call (RPC). In a Microsoft Domain model installation, once the session is

   established between the Continuum Cyberstations Client(s) and the SQL server it is a

   persistent connection once authentication is established.

   In a Workgroup installation, the database connectivity session is re-authenticated and re-established for each database transaction. This results in much slower performance and an inconsistent connection.

9. If you are running a LAN system using Windows 2003 server SP2/R2 and SQL 2005 try disabling the TCP Chimney Offload feature by following the below.
   

   Disable the TCPchimney setting on the SQL server and application server

   Windows 2003 SP2 comes inbuilt with a new network scalability pack which is supposed to accelerate the Windows networking stack. But this setting may not be compatible with all the network cards and it could pose connectivity issues as well. For more information you can refer to

   http://support.microsoft.com/kb/912222

   http://support.microsoft.com/kb/942861

   To disable this feature please take the following steps on both the SQL Server and application server.

   Open up a command prompt on the machine.

   Run the following command: "Netsh int ip set chimney DISABLED"

   Change the values of the following registry keys to 0 a.

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\

   EnableRSS

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\

   EnableTCPA

   Reboot the machine after the above changes.

   ==========================================================

   Disable SynAttackProtect on SQL server: Windows 2003 Sp1 and Sp2 introduces this new feature which monitors processes getting high number of connections and if it detects that too many connections are being made, the connections will be rejected.

   1. Click Start, click Run, type regedit , and then click OK.
   2. Locate and then click the following registry key:

      HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

   3. On the Edit menu, point to New, and then click DWORD Value.
   4. Type SynAttackProtect , and then press ENTER.
   5. On the Edit menu, click Modify.
   6. In the Value data box, type 00000000 . Click OK.
   7. Quit Registry Editor.
10. Sound Card

    In some cases it was found that the built in sound card is causing the problem. If your not using WAV files for alarms then try disabling the integrated sound card in the BIOS.  If WAV files are being played then try disabling the integrated sound card and put in a PCI sound card.

11. There have also been problems with OEM versions of the Operating system of the client machines.  A lot of vendors, like Dell, add additional applications to the systems they ship to assist them if they need to remote into the computer for troubleshooting purposes.  It is strongly recommended that these OEM versions of the Operating System is removed and install the system from a genuine Microsoft installation CD.
12. Go to Control Panel-> Administrative Tools -> services, click with right mouse button Remote Procedure Call -> click properties. remote procedure call properties window is now open, go to recovery and you´ll see: First failure, second failure, subsequent failures -> choose : restart the service. In this setup the system will automaticaly restart RPC when something shuts it down.
13. Check all your Firewall settings.  If possible turn off the firewall for testing and see if the problem goes away.
14. Check to make sure there are no ports being blocked that the RPC Server uses. 

    See KB 908472 (http://support.microsoft.com/kb/908472) details the steps for specifying the RPC port range. 

    Also, be aware of the consequences of reserving the RPC ports. This has the potential to break other applications that assign ports in this range. KB 812873 discusses this, too. http://support.microsoft.com/kb/812873

15. Make sure all Peripheral devices like Network cards, Video cards, Sound Cards, etc. have the latest drivers for them installed.
16. http://support.microsoft.com/kb/913384

    Control Panel > AdminTools >Services and selected the Properties for Remote Procedure Call (RPC)  In the Log On tab, they changed the Log on as: to the Local System Account. It had been "This Account" NT Authority\Network Service.

17. Is an EventNotification configured to send a page on alarm? If a PC running Continuum did not have a modem but on startup Continuum was trying to send an alarm with a page then accdataservice was crashing causing an RPC.
18. Microsoft KB177446 article that describes in depth RPC troubleshooting procedures

    http://support.microsoft.com/kb/177446