Server shuts down even with battery at 100%

[JosepMassagues]

In a customer location, I have an HPE ProLiant ML350 Gen10, behind a SMT1500I UPS.

If I disconnect the UPS, the server keeps on running normally, as it should.

However, evey now and then, something causes the power to fail and the server shuts down immediately.

I suspect that the electrical installation is at fault, and operating heavy machinery may also be involved.

Could some electrical pulse (overvoltage or undervoltage) cause such a catastrophic failure that even the UPS may fail immediately?

The UPS is brand new, manufactured in 2022 and installed in 2023. The self-test come clean.

[Teken · Accepted solution]

I would start with reviewing all the active faults / errors seen by the UPS in the logs. If there is a NMC installed you’ll be able to see more details about the power quality or the lack thereof. 

Next, I would review how the UPS is setup to react during a lights out / grid down event.

You’ll want to confirm Power Chute Serial / Network shutdown software is properly setup too. 

Let us know what the logs show for faults / errors. If you have a NMC installed on the unit please upload both logs to the APC Forum for review.

[JosepMassagues]

Thanks @Teken , no NMC available, only USB. I have just installed PowerChute Serial Edition and started monitoring the UPS on the said server.

I will see what data it can gather that may be useful to troubleshoot the issue.

Still, has anyone any experience on some event knocking down the UPS? What could be the cause?

[Teken]

Appreciate you clarifying it’s the UPS shutting down vs the server. If it was the server my first thoughts would be due to Power Chute Serial / Network Shutdown initiated event. You’ll really want to set it up for the best operational runtime for your environment.

If it’s the UPS and if we ignore any battery related issues for now. From the most common to least common cause are these so go down the list and strike off once confirmed or NA. 

UPS Settings: Verify the system is programmed to resume operations upon power restore vs staying off. Ensure both AVR and Transfer voltages are defined appropriately for the power quality seen in your area.

Serial Cable: If there is a none APC Serial cable attached this can cause the unit to immediately shut down or act erratically. If you’re using a serial adapter do not leave it in place until needed.

Wiring: If the error logs indicate a building wiring fault. This can be related to poor grounding, reverse neutral, more than 5 VAC seen on the neutral, dirty power from a generator, to bad cord set / outlet.

Validate the outlet is mechanically sound, passes basic wiring tests, and measures correctly with a calibrated RMS DMM (Digital Multimeter). Yank on the cord set and wiggle it from the frame to the plug. If nothing happens good move on. If the system shuts off replace the cord / plug.

EPO: The SMT Series UPS does not incorporate an EPO (Emergency Power Off) circuit like the SMX / SRT Series UPS. But, many facilities have a 3rd party EPO in place due to ISO / Insurance standards. Ensure if such a system is in place it’s inspected and validated to work as expected and all wiring and button is 100% sound. 

Battery: The batteries are rated for 3-5 years based on ideal conditions. This is directly impacted by usage, temperature, and environmental conditions. If there is just one weak battery it can take down the entire system! ☝️ Verify the entire battery cartridge is fine from wiring, fuse, connectors, and voltage / internal resistance.

The most valuable diagnostic tools any IT Professional who manages UPS Systems is having a calibrated Fluke DMM, Internal resistance meter, and carbon pile load tester.

As stated earlier it’s important to review and track the operational performance and power quality seen at the UPS. Given there isn’t a NMC installed this will be a lot harder to do over the long term. I do t recall if PCSS / PCNS offers email alerts but if so define it to send you notifications of key metrics and operational states from obvious PCSS / PCNS initiated shutdown’s, AVR activation, On Battery, Site Wiring Faults, etc.

Let us know what you see and observe in the next few days, weeks etc! 👍🍺