Lost communication with APC SMART UPS 1500 (model SUA1500)

[BillP]

Hello

We have more then 400 Windows 2003 R2SP2 servers with all the same image.
It's all IBM X3500 servers (7977-AC1).
The APC SMART UPS 1500 (SUA1500) are connected directly to the server with a USB cable.
We use the Windows agent version 7.0.5.108 and the Infrastruxure Manager Console 4.7 build 275.
In the device manager of Windows, we have the HID UPS BATTERY driver version 5.2.3790.0.

At first, the UPS was communicating normaly and then it lost communication for no reason.
We have about 50% of the UPS with that problem.

1. We known that if we unplug and plug again the USB cable, the communication established.
2. We known that some servers loose the HID UPS BATTERY driver and we found a UNKNOWN DEVICE in Universal Serial Bus Controller. In that case, if we delete the UNKNOWN DEVICE, shutdown server and start the server, the communication established and the HID UPS BATTERY driver appears.
3. We see in the Infrastruxure Manager Console, in the Model Name colum, some UPS in problem with the right model, some with Back-UPS and some with UNKNOWN (normaly in that case, we also se the UNKNOWN DEVICE in the Windows device manager.)

We are asking for you help in that case ...

Tks

[BillP]

.... also, make sure you dont have too many USB devices plugged into the server hogging all the power ....

Respond : We only have the UPS, keyboard & mouse.

[BillP]

how old are these UPSs?

sometimes to relieve communication problems as these, you can try a couple different things.

1.) braindead the UPS - this means doing the following:
-turn off the attached equipment on the UPS and unplug it.
-remove the communication cable from the UPS
-turn off the UPS and unplug it from the wall.
-hold down the off (o) button on the UPS for 3-5 seconds until you hear an audible click and see the LEDs flash.

this procedure resets the UPS communication and can help alleviate some of the issues you mentioned above. you want to make sure too that you dont have the USB cable plugged into an under powered USB hub. on a server, i imagine thats not the case, but i am just throwing it out there. also, make sure you dont have too many USB devices plugged into the server hogging all the power. you can also try changing USB ports.

2.) switch to serial communication if you are able to. i believe this requires a reinstall of the software so most likely you won't want to do it or will want to try it on one machine. then we'd know its something with the usb port on that particular machine anyway.

with infrastruXture manager, it doesnt really help tell me whats going on. ISX manager monitors the PCBE SNMP agent. the ISX manager may lose communication with that PCBE agent and display an error like that too so that could be another issue.

hope this helps to get us started at least!

[BillP]

Oups it didn't show up in my last post :

How we build each server :

We start from a image file and restore it on the server.
We use a APC SMART UPS 1500(SUA1500), i gonna call it APC-X.
After that we ship the server to is final location.
We then connect a APC SMART UPS 1500(SUA1500), but not the APC-X but a new one APC-Y.
At that time everything works fine.

[BillP]

These UPS are brand new (less than a year).

For the braindead, it's gonna be hard to do it. Servers are in many different town.

For now, the option to switch to serial communication, is not a option for us.


How we build each server :

We start from a image file and restore it on the server.
We use a APC SMART UPS 1500(SUA1500), i gonna call it APC-X.
After that we ship the server to is final location.
We then connect a APC SMART UPS 1500(SUA1500), but not the APC-X but a new one APC-Y.
At that time everything works fine.

After many weeks or months, in the event log of Windows, we see suddendly a eventID 3000 Lost Communication with UPS. The server didn't reboot.
No update was made.
We see before that the UPS was operating normaly because it was able to react to power failure.

In a lab, I'm able to simulate the same problem, if a unplug and do not plug correctly (didn't push it until it clic) the USB cable.
Then the UPS stop communicating.

When the problem occurs, there is 2 way to established the communication again :
Remotely : is to delete the HID BATTERY DRIVER or UNKNOWN DEVICE, shutdown the server, restart server.
Then windows is able to detect the UPS again

Onsite : is to unplug and plug the USB cable. Then windows is able to detect the UPS again.

We have try the remotely solution on 5 servers. Out of 5, only one has lost the communication again. But 1 failed again and there is no way that the other 4 won't failed again.

Yesterday, I made a test on a remote server with a UPS that doesn't communicate. It was showing in ISX Manager the right model.
From Device Manager (Windows), i found the USB ROOT HUB that the HID BATTERY DRIVER was using. I then disable it and reenable it.
I made a SCAN FOR NEW DEVICES and it found something but was not able to identify it. So Windows created a UNKNOWN DEVICE.
I then try the same test on a remote server with a UPS that communicate, Windows was able to see the UPS again.

We are still searching and making test to be able to find the possible cause of that problem.
We are not able to tell if the problem come from the UPS, the server or from Windows.

The fact that, at first it's not the same UPS (APC-X vs APC-Y), could it create that type of problem ?
In Windows, is it suppose to use the HID BATTERY DRIVER or the APC BATTERY DRIVER ?
Is there a tool the can be use to monitor the UPS or USB port that could help us ?

Remember that we are working remotely ... if possible we want a remote solution !

tks again

[BillP · Accepted solution]

i believe in windows device manager you should see an APC UPS battery under batteries and then a HID compliant device as well.

i dont know of any USB monitoring tools either to be honest.

on the ones that the USB device is becoming an unknown device would make me think there is an OS issue, especially if you fiddle with it and are able to get it to be recognized again.

i'd really suggest braindeading the UPSs (which I know you said is difficult) but is a sure way to clear up/reset any communication on the UPS if it indeed is a UPS issue.

it sounds like it may be a mix of issues but we could at least work one by one to see if its just a combo of different issues between each of the servers.