PM8000 and redundant ring topology

[PhilipCannon]

I recently ran in to a situation where I was interfacing with some switchgear at a hospital that had PM8000s on each main.  The meters came from the factory networked together via a very basic 5 port Ethernet switch.  Knowing the hospital's IT wouldn't want an Ethernet switch on their network, the contractor ran separate CAT6 connections to each meter.  When it came time to patch them in, they connected to the available RJ45 port and did not disconnect the switch connection at either meter.

Per the hospital's IT department the meters sent 9 GB of traffic in a matter of seconds and ended up taking down 4 of their network switches and the entire hospital wireless.  Fortunately it did not effect any critical systems and in the end they were just relieved to know what caused the problem, but it could have been much worse.  Seeing as the PM8000 supposedly supports redundant ring topology, why did this happen?

[daniel_parker_p · Accepted solution]

Phillip

RSTP can actually be the cause of the scenario you've described if the new loop has not been considered in the existing switch configurations. As a side point, we should be recommending metering/monitoring system be on their own dedicated VLAN (although this may/may not have helped depending on the switch configuration). Also need to keep in mind that when connecting PM8000 to complex networks, Ethernet switches will see them as a switch (not just an Ethernet device), this can cause issues in some cases, in some it may be just that the Ethernet port automatically locks out the PM8000 (e.g. if Cisco BPUD Guard is set up).

[michael_franz]

Please see drawing and see if it helps.  If a true loop was made the IPs would show up twice.  Duplicate address can cause networks to crash.  There are other issues with dual porting (daisy chaining) Ethernet.  If you lose a meter on the string all the units downstream will also lose communication.

Michael

[Eddic]

Agreed!

The daisy chaining on PM8K needs the proper setting on the management switch at both end.

[michael_franz]

Daniel:

Please be aware that many site like hospitals, datacenters and foundries have a difficult time taking outages.

When deploying a daisy-chained Ethernet system it would be prudent to place the meters in a compartment where all dangerous energy can be removed.  When this is implemented correctly meters and communication can be services safely without the need for a substation outage.

Kind of a bit deal.  Many people do not think about this when designing a system.  Hope this helps others.

Thanks,

Michael

[dpearce]

Hi Daniel,

We have also experienced issues with the RSTP setting on the PM8000, we did some testing on the meter with two Cisco switches and found issues with the RSTP setting both enabled and disabled.

Please see the file linked below in relation to the summary of the tests below.

Dropbox - PM8000 Testing v1.0.pdf

In summary

1. Loop created as per the 1^st page
2. No BPDUs seen from the PM8000 on either Cisco switch
3. With STP disabled on the meter the Cisco switches don’t detect the loop (due to BDPUs not passing across the meter) and set all ports to forwarding which causes a loop in the network. Storm control which is configured on the Cisco ports then kicks in and blocks one of the ports to the meter.
4. With STP enabled on the meter again both Cisco ports connected to the meter go into a forwarding state. No BPDUs are detected on the Cisco from the meter. We assume that one of the meter ports is in a blocking state although we have no way to check this on the meter. The network is stable however after a random amount of time a loop will be created in the network and storm control kicks in as per point 3. Storm control is set to auto recover every 15 minutes so at this point we will flip backwards and forwards between the meter putting a port into a blocking state and storm control disabling a port. This is seen in DR10 at the moment we believe.

It would be great to receive your comments.

Regards

Darren

[Darren_Pearce]

Hi,

Just been testing the RSTP setting of a PM8000 meter with the latest firmware v001.004.001  and there still seems to be issues with different scenarios, we can't be the only integrator of meters in switchboards where a redundant connection is required can we?

Could really do with this looking at to resolve fully soon.

Thanks

Darren