Issue
After upgrading systems to the latest I/A Series G3 Niagara maintenance builds 3.5.406, 3.6.406 or 3.7.106 some customers have noticed spikes in Heap and CPU usage. This behavior has been traced to the excessive creation of NiagaraNetwork worker threads.
Customers may see spikes in CPU usage that randomly occur and persist until system appears to halt or in some cases triggers Engine Watchdog Timeout and restart of station. Customers may also experience the Java Virtual Machine exit with the following exception in the standard output:
java.lang.OutOfMemoryError: unable to create new native thread
Environment
I/A Series G3 versions 3.5.406, 3.6.406, and 3.7.106
Cause
It is believed that excessive NiagaraNetwork workers are being spawned due to additional overhead required to encrypt and decrypt passwords from the new password store utilized in 3.5.406, 3.6.406 and 3.7.106.
Excessive usage of CPU and creation of worker threads may be seen on both Enterprise Network Server and ENC/JACE platforms. Typically OutOfMemoryError will be seen on ENC/JACE platforms and not on PC based solutions due to the limited resources of the embedded platforms.
The issue has only been observed on stations with five or more station devices in the NiagaraNetwork.
Resolution
Limit the number of NiagaraNetwork/workers that can be created by setting the property "Max Threads" on the NiagaraNetwork/workers property sheet.
A good rule of thumb for the number of workers is six times the number of Niagara Network stations monitored by the station.
