Subscribing is a smart move!
You can subscribe to this board after you log in or create your free account.
Invite a Co-worker
Send a co-worker an invite to the portal.Just enter their email address and we'll connect them to register. After joining, they will belong to the same company.
Send Invite Cancel
Solved
Go to Solution
Link copied. Please paste this link to share this article on your social media post.
I have 2 ESXi hosts (Host1 and Host2) managed by remote vCenter Server and powered by 2 Smart-UPS RT 2000 RM XL (UPS1 and UPS2) running in Redundant configuration as shown in picture as below. RT1 is router connecting Country_1 and Country_2 networks
During power-outage all VM's and both hosts were shutdown ad powered off successfully. But when external power returned these ESXi hosts did not start automatically as expected. They stayed in power OFF mode even their BIOS is set to be powered ON when external power returns. I checked logs of NMC cards in UPS1 and UPS2. See these logs below. Seems that are differences between some parts of these logs. Both UPS'es were commanded at 11:37:11 to start graceful shutdown process before the UPS turns off. During process UPS1 turned power off at 11:57:39 and then turn it on at 11:57:44. While UPS2 turned power off at 12:04:33 and then turn it on at 12:04:38. It seems that UPS2 was powering both ESXi hosts while UPS1 has been rebooting and opposite, UPS1 was powering both hosts while UPS2 has been rebooting. Because of that external power for ESXi hosts has been always ON and hosts stayed OFF at the end of process, for them power was not lost
Below you can also find PCNS log entries refering to process
Question: why it happened that both UPS1 and UPS2 were not rebooted at the same time even their NMC cards configuration is the same and they are controlled by the same PCNS causing EXSi hosts to stay OFF?
PCNS log
04/07/2021 11:11:50 UPS1 is running on battery power.
04/07/2021 11:11:50 UPS's have switched to battery power.
04/07/2021 11:19:37 Available runtime has been exceeded.
04/07/2021 11:36:50 UPS critical event: On Battery.
04/07/2021 11:37:04 Shutdown sequence started on Hosts Host1, Host2 in response to UPS critical event: On Battery.
04/07/2021 11:37:09 PowerChute is turning off UPS1
04/07/2021 11:37:09 PowerChute is turning off UPS2
04/07/2021 11:37:14 Multiple UPS's have been commanded to turn off.
04/07/2021 11:37:14 UPS critical event: UPS turn off initiated.
04/07/2021 11:37:19 Starting Maintenance Mode Task on Host2
04/07/2021 11:37:19 Shutting down VMs on Host2
04/07/2021 11:37:21 Starting Maintenance Mode Task on Host1
04/07/2021 11:37:21 Shutting down VMs on Host1
04/07/2021 11:37:22 Starting vApp shutdown process.
04/07/2021 11:37:22 No vApp to shutdown on Host2
04/07/2021 11:41:28 Maintenance Mode task canceled on Host1 as there are still powered on VMs. Please verify that sufficient time has been configured for VM/vApp/vCSA VM shutdown duration.
04/07/2021 11:43:07 Host2 has successfully entered Maintenance Mode.
04/07/2021 11:44:54 Shutting down Host2
04/07/2021 11:45:53 Shutting down Host1
04/07/2021 11:46:09 Multiple Critical Events occurred: Low Battery, On Battery.
04/07/2021 11:46:19 UPS critical event: Multiple Critical Event.
04/07/2021 11:46:35 Multiple Critical Events resolved.
04/07/2021 11:46:35 UPS critical event: Multiple Critical Event resolved.
04/07/2021 11:46:46 UPS critical event: Multiple Critical Event resolved on Host2, Host1.
NMC log of UPS1
07.04.2021,13:00:35,System,"Web user 'admin' logged out from qqq.vvv.xxx.245.",0x001F
07.04.2021,11:57:47,Device,"UPS: Returned from bypass in response to the UPS front-panel, a user-initiated software command, or transition to critical bypass.",0x01B4
07.04.2021,11:57:45,Device,"UPS: In bypass in response to the UPS front-panel or a user-initiated software command, typically for maintenance.",0x011C
07.04.2021,11:57:44,Device,"UPS: The output power is now turned on.",0x0113
07.04.2021,11:57:40,Device,"UPS: The graceful shutdown period has ended.",0x014F
07.04.2021,11:57:39,Device,"UPS: The output power is turned off.",0x0114
07.04.2021,11:57:20,Device,"UPS: Started a reboot process.",0x0117
07.04.2021,11:57:09,Device,"UPS: A discharged battery condition no longer exists.",0x0108
07.04.2021,11:56:18,Device,"UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.",0x0107
07.04.2021,11:55:58,Device,"UPS: No longer on battery power.",0x010A
07.04.2021,11:37:11,Device,"UPS: A graceful shutdown process is being used to shut down the load equipment before the UPS turns off. Initiated by PowerChute ns from xxx.yyy.zzz.15.",0x0125
07.04.2021,11:11:48,Device,"UPS: On battery power in response to an input power problem.",0x0109
NMC log of UPS2
07.04.2021,13:00:29,System,"Web user 'admin' logged out from qqq.vvv.xxx.245.",0x001F
07.04.2021,12:04:41,Device,"UPS: Returned from bypass in response to the UPS front-panel, a user-initiated software command, or transition to critical bypass.",0x01B4
07.04.2021,12:04:38,Device,"UPS: In bypass in response to the UPS front-panel or a user-initiated software command, typically for maintenance.",0x011C
07.04.2021,12:04:38,Device,"UPS: The output power is now turned on.",0x0113
07.04.2021,12:04:34,Device,"UPS: The graceful shutdown period has ended.",0x014F
07.04.2021,12:04:33,Device,"UPS: The output power is turned off.",0x0114
07.04.2021,12:04:14,Device,"UPS: Started a reboot process.",0x0117
07.04.2021,11:58:17,Device,"UPS: A discharged battery condition no longer exists.",0x0108
07.04.2021,11:57:52,Device,"UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.",0x0107
07.04.2021,11:55:58,Device,"UPS: A low battery condition no longer exists.",0x0110
07.04.2021,11:55:58,Device,"UPS: No longer on battery power.",0x010A
07.04.2021,11:46:06,Device,"UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon.",0x010F
07.04.2021,11:37:11,Device,"UPS: A graceful shutdown process is being used to shut down the load equipment before the UPS turns off. Initiated by PowerChute ns from xxx.yyy.zzz.15.",0x0125
07.04.2021,11:11:48,Device,"UPS: On battery power in response to an input power problem.",0x0109
Accepted Solutions
Link copied. Please paste this link to share this article on your social media post.
Hi,
The logs show that both UPS received the power off command simultaneously, so we know both UPS received the command. I suggest you check the outlet group power off delay.
Open the NMC web interface and go to Configuration, Outlet Groups
select the outlet group the PCNS client has been assigned to and verify the power off delay is set the same on each UPS.
When PCNS sends the power off signal to the NMC, it counts down the power off delay and then cuts power to the outlet group. If the delays are different, that would explain why one cut output later than the other.
Link copied. Please paste this link to share this article on your social media post.
On 04/15/2021 15:53, Bill said:The logs show that both UPS received the power off command simultaneously, so we know both UPS received the command. I suggest you check the outlet group power off delay.
Open the NMC web interface and go to Configuration, Outlet Groups
Hi Bill,
Thanks for your replay 🙂
I have no Outlet Group in my NMC2 Configuration menu. See picture as below
My both UPS'es are: Smart-UPS RT 2000 RM XL. Do they have Outlet Group feature?
What else can be reason for different rebooting time?
BR, Wojtek
Link copied. Please paste this link to share this article on your social media post.
Hi,
If there are no outlet groups then you should check the low battery duration, maximum required delay, and the shutdown delay. The settings are found under Configuration - Shutdown.
The UPS will wait the amount of time indicated by one of the following, whichever is greater: Low Battery Duration or Maximum Required Delay After this delay, a further non-configurable two minute delay is counted down. The UPS will then turn off after the user-configurable Shutdown Delay time has elapsed.
Maximum Required Delay
- It is the longest shutdown delay needed by any server listed as a PowerChute Network Shutdown client.
- It is calculated whenever the management interface of the UPS turns on or is reset, or when the Force Negotiation option is selected and you click Apply.
Create your free account or log in to subscribe to the board - and gain access to more than 10,000+ support articles along with insights from experts and peers.