Just another case (JAC)

[Anonymous user]

Here is my configuration:
1 x Smart-UPS + NMC
2 x VMware ESX servers with PCNSs
1 x Win2k3 with VMware Virtual Center and PCNS installed
2 x NetApp filers with storage enclosures

ESX servers depends on NetApp storage. NetApp supports UPSs in the way when it sees that x minutes remaining on battery life it initiates shutdown.

I need to shut down all this hardware (and UPS too) immediately after "on battery" event to prevent overheating due to air-conditioners shut off. Here is how I see it:
1. Initiate ESX servers shutdown immediately after on battery event.
2. Empirically calculate remaining UPS uptime after ESXs shutdown (t1). Set this time - 2 minutes at NetApp as shutdown trigger (t2).
3. Set (t1 + 2 (for safety) + 2 (NetApp shutdown) + 2(for safety)) (t3) as shutdown delay in Win2k3 PCNS and also set here to shut down UPS after OS shutdown.
It is approximately 10-12 minutes in total.

The main problem in this scenario is that t2 is difficult to calculate. I see that runtime remining (RR) now is 29 minutes. I heard that RR is slightly different when online and immediately after power failure. Also battery life time will be increased after ESXs shutdown. Skew in t2 also produces skew in t3. And finally I should always recalculate this time when I add new hardware, or load on existing hardware is increased.

Is there any more simple and robust way to do this?

[TheNotoriousKMP_apc]

Vasiliy,

I apologize for the turnaround in receiving a response from someone here on the boards. While it seems you have quite the complicated scenarion on your hands, I did want to have one statement clarified before recommending anything.

You had stated you wanted all of your equipment, UPS included, to power down IMMEDIATELY after a UPS: On Battery event has started. However, you then go into runtime calculations of over 30 minutes to keep some equipment that was mentioned as part of "all equipment" online. If you're worried about overheating since the Cooling units go off, why would you continue to run equipment that's going to generate excess heat back into the room around the products? That seems to contradict what you're trying to accomplish.

Any clarification would be appreciated.

[Anonymous user]

it seems you have quite the complicated scenarion on your hands.

It is more compicated than 1 tier server configurations, but I think that it is frequent scenario in IT environment when servers are mixed with storage.

IMMEDIATELY after a UPS

Yes. I'm starting shutdown immediately after "on battery" event as 1 step states. But servers mount LUNs from storage, so I need to assure that servers end all their I/O activities with storage prior to storage shutdown. So firstly I shutdown servers and only after that I shutdown storage. Their is one more appliance - Win2k3 server. It manages ESX servers cluster but doesn't depend on storage. I decided to shutdown UPS from this very server. Maybe it is unnecessary but I don't know another way to shutdown UPS in this case.

I hope that it makes more sense.

[BillP]

Unfortunately, to myself, it seems like a "skew" that is unavoidable. I cant think of another way to do this personally other than calculating and testing 😕

[Anonymous user]

it seems like a "skew" that is unavoidable

I see that all more or less complicated UPS shutdown scenarios are based on empirical delay calculations. But maybe there is more simple and robust way to do this in my case. I have found a tricky way to do this. Two NetApp filers are in High Availability cluster. So when one filer fail then another failover without data loss. So the sequence can be such:

1. Receive "on battery" event. Start shutdown of ESX servers and when ESX_1 came to the end of shutdown sequence (I hope that it ends all its I/O at this moment and unmount all LUNs) issue shutdown command of Filer_1 via LAN.

It doesn't matter if ESX_2 ends its I/O because Filer_2 failover at this moment and all goes well. When ESX_2 ends its shutdown sequnce it issues shutdown of Filer_2.

2. Calculate overall time of first step and set it as shutdown delay for win2k3.


But I don't know if first step is feasible. I should consult VMware specialists.

Actually all this ways doesn't inspire me with confidence. Any changes in configuration or datacenter load can result in data loss due to broken sequence. So UPS becomes not a defender but potential threat to data integrity.

[Anonymous user]

I still hope to see any recommendations from The Notorious K... 🙂

[TheNotoriousKMP_apc]

Sorry that you were expecting a reply from me. Angela jumped in on this and I thought that she provided good enough information to relay.

As I stated before, there's only a Tier 1 shutdown available for PCNS and UPS. Outside of command files, there's no way to do any other types of shutdowns, and all of these would have to be tested in your scenario, which is dealing with a non-linear load, so the calculations that you get the first time, may not be the same the second, third or fourth.

I apologize that there's no easy solution for this. But the APC Software doesn't really go beyond a Tier 1 setup.

[Anonymous user]

Thanks for your reply Kevin.

But your answer is totally unclear for me.

there's only a Tier 1 shutdown available for PCNS and UPS
So firstly what do you mean 'Tier 1' here? Tier 1 = servers? Tier 2 = storage?
Secondly I think that term 'tier' is not appropriate here. UPS and PCNS don't deal with tiers at all. They deal only with configurable delays and appliances where PCNS is set up. If you mean that I can't shutdown storage from PCNS, surely I know it.

Outside of command files, there's no way to do any other types of shutdowns
Do you mean with this phrase that the only way for me is to write a command file on ESX servers to shutdown storage?

But the most confusing thing for me is: *"Why there is no standart solution for this scenario when almost all mid-range to high-end datacenter designs apply storage?"* What do they do? Hire night technician?

[TheNotoriousKMP_apc]

Harmont wrote: So firstly what do you mean 'Tier 1' here? Tier 1 = servers? Tier 2 = storage?

Outside of command files, there's no way to do any other types of shutdowns
Do you mean with this phrase that the only way for me is to write a command file on ESX servers to shutdown >storage?

But the most confusing thing for me is: *"Why there is no standart solution for this scenario when almost all >mid-range to high-end datacenter designs apply storage?"* What do they do? Hire night technician?>

Tier 1 is just Server interaction with PCNS. I thought you stated it was 1+ Tier in a prior post.

Outside of command files, there is no way to shutdown anything without an OS capable of doing a shutdown.

As for why there's no standard? I'm not sure. But because the software can only shutdown an OS, I believe that's why we made the command file an available option.

[Anonymous user]

As for why there's no standard? I'm not sure. But because the software can only shutdown an OS, I believe that's why we made the command file an available option.
Maybe I understood something wrong but command file is not an option in my situation. ESX server interact with storage even if all VMs are shut down. AFAIK script executes before initiating OS (ESX in my case) shutdown. So if I'll execute script which should shut down storage before storage is logically detached from ESX (I suppose it occurs at some of the last steps of OS shutdown sequence) I'll face data corruption.

So the only way is to implemet custom shell script and insert it at the end of ESX shutdown process. Could you confirm or deny my assumption?

[Anonymous user]

OMG, I suppose I have invented a solution.

1. On battery event occured. Start ESXs shutdown with 1 min delay for example (to prevent shutdown in the case of short power outages). At the end of each ESX shutdown sequence initiate corresponding storage shutdown (ESX 1 shuts down Storage 1, ESX 2 shuts down Storage 2). Disappearance of Storage 2 to ESX 1 (and vice versa) is not a problem 'cause storage is in active/active cluster. I mentioned it before.
2. The main poin here is not to shutdown management server immediately. I can shut down it after UPS battery is totally exhausted. There is nothing really hot there in contrast with storage enclosures. So I don't need to calculate all these 'one delay plus another and so on'. When battery is exhausted I shut down management server + UPS.

But there is another problem. What if power is restored after servers and storage is shut down but before management server becomes down? I suppose that management server shutdown is canceled but storage and servers left in down condition.

[TheNotoriousKMP_apc · Accepted solution]

Hi,

Let me first apologize. I do not posess the knowledge about your setup of servers with data storage seperate to be able to fully assist you in this manner. That is why my recommendations are so skewed.


As far as your last question, if a shutdown is initiated and power returns to the UPS, it will continue with the shutdown, then reboot itself and bring everything back up (the catch-22 there is if the bios and power supplies are configured to power up upon having power restored to them).