Issue
bCX going offline and logging multiple 0x1408s in its error log.
Environment
bCX4040
ALL CX master controllers.
Cause
Excessive traffic on the network causes controller to run out of message buffers.
Every time a controller receives a communication packet it has to allocate a block of memory (a buffer) to place the packet until it can be processed. Under normal network conditions the packets are processed at a rate that exceeds the rate at which they are coming in, in some cases typically associated with network configuration issues or malfunctioning nodes on the network, (i.e. broadcast storms) the firmware in the controller is unable to keep up with the amount of packets being received which exhausts message buffers available and renders the controller unable to communicate properly.
In many cases the condition is temporary and does not result in a major problem, it is not unusual to see sporadic 0x1408s in the error log but when the condition persists for longer periods it will have an impact on the ability of the controller to communicate properly on the network.
Resolution
Identify and remove the root cause of the excessive traffic.
In the WireShark capture above, a constant WHO-IS sent out by BBMD devices due to part of the network being disconnected was causing a "WHO-IS storm" resulting in the bCX controller on the subnet to go offline and register several 0x1408 errors per minute.
