Potential Network broadcast storm detection alarm on BMENOP0300 card

[Maitree_ri]

Dear Community

          Our M580 system have alarm "Potential Network broadcast storm detection" alarm on BMENOP0300 card. As I try to checked that cause from  we changed polling from %S7 to %S6. I use this value for polling digital input data and measurement data from  P5F30 Relays. They connected ring loop communication. Please see my question as below. 

          1. Could you advised how to config the system to no alarm

          2. Could we ignored this alarm and block red indicator to show on BMENOP0300 card.

          3. Could we separated digital input point with %S4 and measurement value with %S6 and no any alarm

          4. This alarm above, do we need to setting at Easergy P5 relay side.

Best Regard.

Maitree Riyaud

[Siddiqui]

Hi @Maitree_ri 

In my opinion, a proper investigation and finding the root cause is needed for the Potential Network Broadcast storm issue.

Can you share the network architecture to get a clear picture, at least how the things are connected between this particular NOP module and the IEDs?

My understanding is that when you changed the polling interval from 1 minute to 1 second, the issues started, correct? So, if you changed the polling interval back from seconds to minutes, was the issue resolved automatically? It is essential to understand whether the problems are coming from the polling interval only or from somewhere else.

I suggest you collect the Wireshark logs from the BMENOP0300 module Service port 1 and configure this service port as a mirror port for ports 2 and 3 using the Modicon IEC 61850 configuration tool to find the root cause while analyzing the Wireshark logs.

• For the first and second queries, my recommendation is to find the root cause and fix the issues instead of trying to hide or block/ignore them at the module level.
• For the third question, yes, you can have a different polling group. You can group digitals in one group and analogs in another group using the Modicon IEC 61850 configuration tool.
• For the fourth question, we have to find the root cause first before changing the settings at Easergy P5.

What kind of mechanism is used to exchange information between the BMENOP0300 module and the IEDs? You mentioned only polling. What about reporting? Are there any GOOSE messages exchanged between the IEDs or between the IEDs and the BMENOP0300 module?

Are you checking the feedback status of the polling command before initiating the following polling command? Also, check the BMENOP0300 module device DDT for the broadcast storm information.

What about the first BMENOP0300 module? Is this module facing any issues?

I need the above information to provide help, but at least the above comments will help you start the investigation.

Best Regards,

MWS.

[jaysonlkh]

Based on the error message, it seems like you are linking the backplane with the M580 CPU with the BMENOP0300 module. Then you have to consider to enable the IP forwarding feature within the BMENOP0300 and check the overall network architecture to ensure there to have correct setting of RSTP.

[Maitree_ri]

Dear @Siddiqui

         See system architecture as attached

         As your understood is corected at polling interval 1 minute do not have alarm, at interval 1 second has alarm but not imediately. The alarm has occured when pass 1 or 2 days later.

         For wireshark log, we can't not do it. because of it need to configue port but the customer has used the PLC for theire plant process

        For mechanicsm exchange with IED, the current we use polling, How to set reporting at relay EASERGY P5, please advised

        Yes  we are checking the feedback status both 1 minute and 1 second do not have difference to verify the alam cause from time interval.

        First BMENOP0300 don't have alarm in time interval 1s

Dear @jaysonlkh 

         We do not enable forward port in backpane, About RSTP issued, we connected and setting communication cabel same as BMENOP0300 first modules. if you have any advised for setting. let us know.

Maitree R

[jaysonlkh]

After checking the system architecture, I have a doubt that CPU module Port 2 is being connected to the upper switch. I will recommend to interconnect between CPU A and B and close the RSTP loop with your CRA modules. And let see this will improve and resolve your issue.

[Siddiqui]

Hi @Maitree_ri,

Thank you for sharing the architecture, and check my comments below:

• First of all, I would recommend that you create a case with the support team to have a proper follow-up, if you have not created the case with them.
• Correct my understanding: Does the second NOP module enter this error state randomly, get out by itself, and then again enter this error state? The module health status is monitored at the SCADA level, which is how the operator discovered that something was wrong with this module.

It would be great if you could also provide the exact PLC in the shared network architecture where you are facing the issue. Anyway, I managed to detect it based on your shared Excel sheet with the ECE program name:

Is this the correct PLC where you are facing the issue, as shown in the picture below?

I want to clarify the following questions:

• Where is there a need to change the polling interval?
• The Network architecture dates back to 10 October 2024. Has this issue been present from the start, or has it just appeared now?
• Are you sure this drawing is the latest and updated one with all the correct physical connections? Because the following PLC does not show any RIO connection?

• If we go back to the selected PLC, as mentioned by @jaysonlkh, the RIO loop is closed by using the SW1 and SW2. Is there a reason for using these two switches? What kind of switches are these?
• Why are only the primary rack's NOP modules service ports connected with SW1 and SW2? And not the standby NOP module? And for what purpose?
• Are only two P5 relays talking to each NOP module in this configuration?
• Verify and confirm that the P5 relay and NOP modules got a direct connection without any switches or devices in between?
• Did you configure the proper RSTP configuration among the NOP modules and two P5 relays?
• You can use the NOP module DDT to detect the RSTP role of Port 2 and Port 3, as well as the Easergy Pro tool to detect the RSTP diagnostics for each P5 relay.
• Based on the provided architecture, it seems like the RIO network with two remote drops and the two primary rack NOP module networks are connected using the SW1 and SW2. Can you clarify this?
• You are facing issues only at the primary rack NOP module?

Moving forward, my advice is as follows:

1. First, clarify how each module is connected physically and why. Because there are discrepancies based on the shared architecture, you have to verify physically first how the systems are connected.
2. Verify that each BMENOP network is physically isolated from the rest of the network. Which is not clear with the shared architecture because the service ports of only the primary rack NOP module are connected using the switches?
3. I don't suspect the issue is coming due to the polling interval, as you already mentioned that you are using the polling command status feedback before issuing the following polling command.
4. The issue is coming at layer 2 of the Ethernet. After verifying the physical connection and, if needed, performing the correction, the best way is to capture the Wireshark logs. You can use the hub device between the NOP module ports and the P5 connections and capture the logs to know what exactly is happening at the wire level instead of guessing.
5. Regarding the use of reporting, first, you have to configure them inside the P5 relay and then configure them inside the NOP module. I would recommend you follow the TVDA to understand how reporting works, but I don't recommend making any changes without first finding the root cause. Connect Modicon M580 to an IEC61850 Network
6. IEDs are there to provide protection and might exchange time-critical information between themselves, not just respond to the polling request of the NOP module. That's why it's essential to know what is at the wire level in the NOP module network besides polling requests.

I hope the above comments will help you to start the investigation.

Best Regards,

MWS

[Maitree_ri]

Dear  @Siddiqui 

       Thank you for your support.

       We are trying to investigated the problem

       Please find my reply as attached.

Maitree R.

[Siddiqui]

Hi @Maitree_ri, 

You are welcome. Thank you for your response. I have already taken a look at your comments. I suggest you ping me on Microsoft Teams. You can look for my name to find me. This way, I can respond to your queries and suggestions easily and more quickly, instead of going back and forth a couple of times.

Once we find the root cause and fix the issues, we summarize the findings here that would be helpful for the rest of the community.

Best Regards,

MWS