Does POE reset PM8000 meter and possibly PM8ECC?

[sesa238193]

I have a customer who has had sporadic issues with two PM8ECC's at their site.  For some unknown reason, the ECC cards would drop communication.  Looking at the logs, I sometimes saw that a power up / reset event was captured during that time, but communications were not restored.

Yesterday, someone was in the field to replace the PM8ECC's with PM8000s.  He had the PM8000 hooked to his computer over Ethernet, and the meter functioned just fine.  However, as soon as he plugged that meter into the customers network, it started power cycling.  At least 7 times, probably within 5 minutes.  And then continued to do so.

We suspected, and confirmed that all of the network drops for this customer had Power over Ethernet (PoE).  Latest report is that the customer shut off the PoE for that connection, and the meter stopped power cycling.  We won't know for sure that this is the culprit...until we can verify it doesn't randomly power cycle for a long period...months.

Has anyone experienced this before, and has the product team tested what happens when an Ethernet connection with PoE is plugged into the meter?

Best regards

[Bojan]

Hi Jesse,

I had a similar situation with EGX connected to the POE switchboard port - it is continuously “flapping” until the administrator configured the port so that the POE is disabled:

Feb 26 11:07:49.541 EST: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:07:48.965 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:07:49.804 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:07:53.173 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:08:00.046 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:08:00.994 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:08:02.308 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:08:04.497 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:08:11.527 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:08:12.022 EST: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/35: Power granted (SOUTH0-3)
Feb 26 11:08:12.500 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:08:13.775 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:08:15.796 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:08:22.894 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:08:23.439 EST: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/35: Power granted (SOUTH0-3)
Feb 26 11:08:24.144 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:08:25.443 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:08:27.447 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:08:34.948 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:08:35.921 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:08:37.203 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:08:39.216 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:08:46.088 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:08:47.044 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:08:48.368 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:08:50.381 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:08:57.295 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:08:57.866 EST: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/35: Power granted (SOUTH0-3)
Feb 26 11:08:57.983 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:08:59.264 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:09:01.327 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:09:08.936 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:09:09.917 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:09:11.200 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:09:13.230 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:09:20.453 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:09:20.923 EST: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/35: Power granted (SOUTH0-3)
Feb 26 11:09:21.921 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:09:23.221 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:09:25.242 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:09:31.870 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)
Feb 26 11:09:32.432 EST: %ILPOWER-5-POWER_GRANTED: Interface Gi3/0/35: Power granted (SOUTH0-3)
Feb 26 11:09:32.827 EST: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi3/0/35: PD removed (SOUTH0-3)
Feb 26 11:09:34.117 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to down
Feb 26 11:09:36.147 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/35, changed state to up
Feb 26 11:09:42.960 EST: %ILPOWER-7-DETECT: Interface Gi3/0/35: Power Device detected: IEEE PD (SOUTH0-3)

Best regards,

Bojan

[sesa238193]

Was that an EGX300 or an EGX100?  I've heard of it happening on an EGX100...but it was my impression that the EGX and PM8000 were completely different families.

[Bojan]

It was EGX100 powered by 24Vdc Control power, so the POE switchboard port shouldn't detect any Power Device, as in your case.

[sesa42108_bridg]

I have experienced that myself with an EGX100 that had both 24V DC control power, and a PoE connection.  The device would keep cycling unitl one or the other was disconnected, then it worked fine.

[sesa42108_bridg]

For anyone reading this thread that may have some experience with the PM8000 meters, would you have any other theories of what could cause the PM8000 to reboot periodically?  This device was just installed yesterday, out of the box.  The control power is connected to a UPS that was delivering 119V AC continously, and there were two other meters(PM8's) in the room being fed off the same control power and they were not cycling, so it has us scratching our heads.

[BrianB]

John and Bojan.  Can you please give me the firmware versions your EGX's were running when you experienced the issue with PoE.

And I do believe there is no relation between the EGX comms and the Pm8000.  For this I would contact tech support and start the process of investigating.  If they are unsuccessful it will be escalated up to L4 (which is my Role) and I will assist with troubleshooting it along with Engineering.

Brian

[Bojan]

Hi Brian,

Gateway Information

Firmware Version: 3.000

Serial Number: 33127764

Model Number: EGX100SD

Hardware Version: G2

Manufacture Date: 2014-01-18

Best Regards,

Bojan

[BrianB]

John, What type of Ethernet cable are you using, is it a shielded cable?  If you could grab the part number of the cable you are using we would like to do some tests here at our facility.  In parallel please open a tech support case.

[James_Clark]

Can we please clarify if the meter being installed is a PM8000 or a PM800?   I noticed PM8ECC mentioned which is used with PM800, not PM8000.  

[sesa238193]

Hey James.  A PM800 with an ECC is being replaced with a PM8000.

[sesa42108_bridg]

I don't have that info available right here, as it was a device onsite somehwere, at this point I don't even remember where the site was.  It was a new unit, so it should have had the newest FW.  I would say it matches up with the info that Bojan sent.

[sesa42108_bridg]

The meter that is cycling is a PM8000, non RD unit.  I removed a PM800 with an ECC card and put this PM8000 in it's place.  Like Jesse has showed, the meter just continiously power cycles. The whole reason we were replacing this meter was due to the PM8 with ECC was locking up commumications through it, with some power cycling also. The hope was that it was an issue with this PM8 meter and/or ECC, and by replcing it, the issue would go away.

[sesa82206]

I am aware of two other customers where PM8 meters with ECCs are cycling power as well. Is there anything common between PM8000s and PM8s (any component per say) or something that could cause such similar behavior?

adding with support

Regards,

[sesa82206]

By the way, when I said PM8s I was refering to PM800s resetting on their own,

[joshwood · Accepted solution]

Tech Support has the case now.

Just to recap on testing we have done.

We attempted to use a POE injector on a Pm8000, the injector detected that the device did not support POE and did not apply power, this was indicated my a fault light on the POE injector as detailed in the product manual.  We then verified a switch port in house had POE enabled by powering up on a a COmX200 plugged it directly into a switch.

The COMx Powered on , so we plugged it into the Pm8000 and verified coms through PME.  The meter has yet to power down, The switch is a Cisco 2960-x, The pm8000 is being powered by 120v Single phase.

We will continue to monitor for reboots.

[James_Clark]

Team,

I connected a PM8000 to two different PoE injectors.  One from PowerDsine (company that certifies PoE devices) and one from Phihong.   I have have not observed any issue with the PM8000 meter while connected to either of these injectors.   Keep in mind that PoE injectors are considered mid-span devices and inject power, as needed, on the unused pins in an Ethernet cable.   A PoE enabled switch is considered an end-span device and injects power on the data lines directly, as needed.    Unfortunately, I do not have a PoE enabled switch to test this configuration.