AP9630 issue with SUMX6.06 and AOS6.06 Bootmon V1.02

[Anonymous user]

I have an SMT750 with firmware version UPS 05.0 (ID17).

Just updated the firmware to the latest, from something older ..... that shipped with the SMT. The AP9630 card is working properly with one obvious exception. The "Quick Status" initial screen has the graphics, but no data. Blank. (screen shot attached) Nothing in the logs showing an error. All the other screens have the correct voltage, power consumption, data, etc. I also had to re-enter the e-mail server information, as that did not transfer.

Any idea why this is happening? How do I fix? Also, is there a way to update the firmware on this SMT?

Any and all help and advice is appreciated!

Thank you!!

Charlie

[BillP]

Hi again,

Can you go to About->Support in the web UI of the management card and do the generate logs and download logs functions and attach those files here? I have a feeling that UPS->NMC communication has really been lost and the "online" on the status is more of a false positive indicating communication is present. The logs will help me in confirming further since I could be wrong going by what you said that other screens are providing accurate information.

I also found recently when I did an upgrade that the mail server information did not port over properly.

For the UPS itself (ID17), there is a new update available which is UPS 07.1.

Here is the article I created and how to use your AP9630 to push an update to your UPS-> How do I upgrade my SMT/SMX Smart UPS firmware using my AP9630/AP9631 Network Management Card? | FAQ...

Let me know if you need help after reviewing it. You'll need to visit another article to download the UPS firmware file (.enc file called SMT17UPS_07-1_Le.enc) and then use the instructions I've provided with that file.

[BillP]

I deleted your other post after I downloaded the file. To attach a file, where you type your reply, make sure to click "use advanced editor" on the top right of this box and it gives you the option to attach files. You can then selt that as the default if you want.

The UPS update available does not really impact the issue you're seeing so I mean if you want the latest and greatest, update, but if everything is working fine, you can skip it.

Ok, the logs are strange which what you're seeing is also strange. The data log is populated with values but the event log shows no UPS events - did you manually clear that today or did it clear on its own?

Anyway, the first thing I would do since its the easiest is reboot the management card. If you can do it, I'd just walk over to it and reseat it in the management card slot which is OK to do and does not affect the load on the UPS. If you can't do that, go to Control->Network and Reboot the management interface of the AP9630 See if any change there.

If it does not change, my next thought is to reset/braindead the UPS and check. To do that, you would need to turn off and disconnect any attached load on the UPS and I'd also go ahead and unseat the AP9630. Then, turn off the UPS itself and unplug it. After that, you'd have to disconnect the battery which it might work if you unplug the battery disconnect on the rear of the unit (triangle rubber shaped thing in the bottom left corner). I typically disconnect the battery but I am not sure if you'll be able to or if you need a screwdriver. Then, you hold down the off button for about 10-20 seconds and see if you hear any clicks or see anything happen. After the process completes, go ahead and reconnect the battery/battery disconnect and insert the management card back in. Then you can plug the UPS back in and turn it back on and reconnect your equipment and see if any status change in the management card.

Let me know if you have questions before you get started. I'll also see if I can attach the file to your post that I downloaded.

[Anonymous user]

Angela N. wrote:

I deleted your other post after I downloaded the file. To attach a file, where you type your reply, make sure to click "use advanced editor" on the top right of this box and it gives you the option to attach files. You can then selt that as the default if you want.

The UPS update available does not really impact the issue you're seeing so I mean if you want the latest and greatest, update, but if everything is working fine, you can skip it.

Ok, the logs are strange which what you're seeing is also strange. The data log is populated with values but the event log shows no UPS events - did you manually clear that today or did it clear on its own?

Anyway, the first thing I would do since its the easiest is reboot the management card. If you can do it, I'd just walk over to it and reseat it in the management card slot which is OK to do and does not affect the load on the UPS. If you can't do that, go to Control->Network and Reboot the management interface of the AP9630 See if any change there.

If it does not change, my next thought is to reset/braindead the UPS and check. To do that, you would need to turn off and disconnect any attached load on the UPS and I'd also go ahead and unseat the AP9630. Then, turn off the UPS itself and unplug it. After that, you'd have to disconnect the battery which it might work if you unplug the battery disconnect on the rear of the unit (triangle rubber shaped thing in the bottom left corner). I typically disconnect the battery but I am not sure if you'll be able to or if you need a screwdriver. Then, you hold down the off button for about 10-20 seconds and see if you hear any clicks or see anything happen. After the process completes, go ahead and reconnect the battery/battery disconnect and insert the management card back in. Then you can plug the UPS back in and turn it back on and reconnect your equipment and see if any status change in the management card.

Let me know if you have questions before you get started. I'll also see if I can attach the file to your post that I downloaded.

Angela,

1. Thanks for the "advanced" tip. I would have never guessed that was the trick to add a file.

1.5 I removed the SMT750 from service and subbed an older SUA unit while I test/configure. Now in my "lab" environment.

2. I tried everything. First I removed the 9630. Second, I powered down the SMT750 via the power button. Third, uplugged 750 from wall. Fourth, disconnected the battery pack by physically pulling forward out of it's connector. Finally, I held the power button down for 30 seconds.

3. Put it all back together and powered back up. Same results. Quick Status is void of all dynamic data . . . only graphics.

Yes, I did clear the logs a number of times to make it easier to see anything that might be added.

So where do I go from here? Any other ideas to get this fixed?

Charlie

[BillP]

Can you try the AP9630 in the SUA and see what happens? You can just pop it in there. That can tell me if it is following the UPS or if it is the NMC.

Beyond that, then I may suggest we "format" the NMC and wipe its flash memory and perhaps re-flash it depending on your results of what I just asked. That can be done by issuing the format command from telnet/ssh or holding down the pinhole reset button for 20+ seconds while watching the light flicker green and changing to orange to indicate it is successful.

I haven't seen this one and I tend to think its the NMC but i rarely say that since it is a product I work with 24/7

If that does not work, I will escalate accordingly. Regardless, I'll get you a solution. No worries.

[Anonymous user]

Can do. Will put 9630  in a SUA and report back. If need be, then reformat and reload.

Hey, good thing "they" were wrong about using a 9630 in the SUA 🙂

Charlie

[Anonymous user]

Angela,

Well, attached is a screenshot of the AP9630 in a SUA1500.... "The local network management interface-to-UPS communication not established".

Rebooted the card several times.

So, I assume the next step is to format the card and reinstall the firmware?

Can you please advise the process if that is what you want me to do???????????

Thanks

Charlie

[BillP]

Was this ever working? Can you give me some background on what led up to this issue today?

Yes, go ahead and format it. So you reset the SMT750. For sanity's sake, can you also do that procedure on the SUA? It will be a little different as you don't need to remove the battery. You can unplug load, unplug UPS from all and hold down the off button (o) on the front and wait for LEDs to flash and and an audible click.

Lastly, just so you're prepared, a format will wipe out every configuration setting (but will not downgrade firmware) so you'll need to reconfigure the TCP/IP settings later which hopefully you can do.

[Anonymous user]

Everything was working properly, 100%, until I upgraded to the latest SUMX 6.0.6 and AOS 6.0.6.

I pin-holed the 9630 in the SMT750 and got the yellow-green alternating LEDS on the RJ45. Reconnected to LAN and got the same results with the Quick Status.

What exactly would you like me to do at this point?  I would prefer to leave the 9630 in the SMT750 for further testing. Remember, I have already brain-dead the SMT750 earlier today with no good result. If you really think it makes sense for testing, I can move the 9630 into a SUA1500 we have as a spare, but that doesn't make a ton of sense to me, since the 9630 will be going back into the SMT.... so why not get it right in the SMT, first?

I don't mind reconfiguring the TPC/IP and everything else. I assume I can use the APC utility to ID and find the unconfigured/reset 9630. Right?

What is the process (step by step for idiots like me) for doing the format?

Charlie

[BillP]

Ok, let's focus on the SMT750. You can put it back in there. Again, I was just trying to see if the card could talk to another UPS.

To format, the easiest way is just to log in to the Command Line Interface (via telnet/ssh/serial cable) and type format, type yes, to confirm that, and then reboot. You'll also need to type yes to confirm the reboot. The alternative way is to hold down the pinhole reset button for 20+ seconds. It sounds like you just pressed it once and didnt hold it which is essentially just doing a simple reboot of the management interface. If you choose to hold it down for the 20+ seconds for formatting it, you need to ensure you keep it steady and while its held, you'll see it pulse green and change to orange when complete for the status light. If you do not see this happening with the status LED, it's not being properly for one reason or another.

And yes, to reconfigure, you can use the same methods as when the card was new, including the device IP wizard.

If a format achieves no different results, my other thought is to downgrade the card to firmware v5.1.7 via FTP. If we need to do that, I can get you the required files to downgrade.

If that works, I will be extremely surprised and want to look into what could cause that. This is what I would be doing if it were my card. I don't see how a firmware upgrade would cause this so its a good learning experience for me in case it happens again and obviously to fix your issue. The worst case, if for some reason this is a coincidence, then we'd look into replacing the card if everything is pointing to it versus the UPS.

[Anonymous user]

I will try the format tonight or early AM tomorrow and advise the results.

It is a real pleasure working with you!!!!!!!!!!

Have a good evening.

[BillP]

Sounds good.

[Anonymous user]

I just upgraded the SMT to the new firmware.... With the hope that it would help. You were right, no change.

Formatting the NMC early AM. I assume I just run the EXE to reinstall 6.0.6  post format?

I gotta ask...... After I made such huge deal today over RI support. Where are you? It is after 9PM here!!

[BillP]

I am in RI myself. Work never stops!!

I would just format the NMC at 6.0.6 and check it right after and if that doesn't work, we will get the .exe for 5.1.7 and downgrade it via FTP and see what happens. (The 5.1.7 .exe does not allow a downgrade so that is why you need to do it manually via FTP which we have instructions for). If that does not work, I may just suggest you visually verify the card for no obvious damage and I'd just want to see the event log one more time after 5.1.7 and if it is still not working, I will probably replace it and get it back to my desk and take a look at it.

[Anonymous user]

Wow, working very late! I knew that had to be RI!

I formatted the 9630 in the SMT 750. Hyperterm'd back into the card, set the IP address and then used the wizard to reinstall firmware 6.0.6. Last night I also updated the SMT750 firmware.

Funny results with 6.0.6. I had been using the same work station for all my work (WIN7 - IE 10). At first I got the same results on my work station after the format and reload; the Quick Status had no dynamic data, just graphics. I was thinking maybe it was a browser issue, so I tried logging in from 3 different computers with IE 10, Chrome and Firefox.  NONE of the other three would even start with the "Quick Status" screen.... just the home screen. I logged off each time, waited 5 minutes and logged back in. In all three browsers, on all three machines, I got the same results. . . .  no Quick Status (See attached screen shot and note the gray line "Smart-UPS 750:Unknown"). I then went back to my original work station and deleted all the IE temp files..... On this machine I still DO get the Quick Status, but with IE10 only. It is still devoid of data. I am VERY confused at this point.

I am wondering if this is an issue with the embedded web pages in the upgraded 9630? Does everyone else get the Quick Status without issue using 6.0.6?

And our next step?? Downgrade the firmware? Please send the file and instructions.......

Thanks for sticking with me on this.

Charlie

[BillP]

Ok, the attached screenshot looks good and appears to be displaying proper information . Verify under Status->UPS that everything is showing because it looks like we have UPS communication now. It also says No Alarms Present, UPS online which is norma. You can also show me the latest events under Logs->Events->log to verify further as well but the most recent event in that screenshot at 9:39 shows UPS->NMC communication establishhed as it should be.

Unknown after Smart UPS 750: on the top there is just the UPS location programmed in (that you lost when you formatted the card). My UPS also says the same thing currently cause I never configured it. You can see this by changing the value from unknown to something else under Configuration->General->Identification.

Anyway, as far as browsers, I've tried Chrome, Firefox, and I personally have IE9. I have not seen issues with this quick status page - only with other items where I've put IE into compatibility mode and it works fine after. For instance, the data log column headings not being aligned correctly is an issue I've seen.

This looks promising based on what you've provided so far so I am thinking we have fixed it as I type this.

[BillP]

Hi again.

Something has changed between your post earlier today and this latest post. The two screen shots are definitely different. The one earlier today from 9:43 looks normal to me and is what it should look like where as the one with the missing information seems like a place holder of some sort that is showing when the NMC has no communication to the UPS. It looks like the older style screen from v5.1.X.

After looking more closely at my SMX1000 UPS, the normal quick status screen looks like your 9:43 screenshot (no bar graphs and missing information). See below:

See what I mean?

When I look at your event log, everything was OK and then something broke. See below where we have good communication:

07/17/2013 09:39:47 Device UPS: Restored the local network management interface-to-UPS communication. 0x0101

07/17/2013 09:39:35 System Network service started. System IP is 192.168.1.240 from manually configured settings. 0x0007

07/17/2013 09:39:29 System Network Interface coldstarted. 0x0001

07/17/2013 09:38:31 Device UPS: The output power is turned off. 0x0114

07/17/2013 09:38:31 Device UPS: The power for the main outlet group, UPS Outlets, is now turned off. 0x0135

07/17/2013 09:38:21 Device UPS: The power for the main outlet group, UPS Outlets, is now turned on. 0x0134

07/17/2013 09:38:20 Device UPS: The output power is now turned on. 0x0113

07/17/2013 09:38:17 Device UPS: Main outlet group, UPS Outlets, has been commanded to turn on, sequenced. 0x0174

07/17/2013 09:38:12 Device UPS: The output power is turned off. 0x0114

07/17/2013 09:38:12 Device UPS: The power for the main outlet group, UPS Outlets, is off. 0x0186

07/17/2013 09:38:11 Device UPS: Restored the local network management interface-to-UPS communication. 0x0101

07/17/2013 09:37:59 System Network service started. System IP is 192.168.1.240 from manually configured settings. 0x0007

07/17/2013 09:37:53 System Network Interface coldstarted. 0x0001

So its making me wonder basically what happened after ~9:45ish since it seems like communication has been lost after that. The data log looks OK the whole time between since then and now.

I also need to find out why this place holder screen is coming up with the bar graphs because they should not be showing as part of our 6.X.X interface I just verified.

Anyway, am I making sense? The bar graphs should not be showing and it seems like someone did not remove them from the interface as they should've because its displaying the incorrect screen, including the bar graphs, when your UPS is not talking properly to your management card. The screen you sent me time stamped 9:43 is what your quick status should look like for v6.0.6 firmware and I will see what I can find out about getting this addressed for the future version because its misleading.

So to me, we are back to finding out what is causing your management card to lose communication with your UPS. Did you connect any serial cables or anything after doing the format and all that? Or mount anything on top of the UPS at all?

[Anonymous user]

Wow. I haven't had this much fun since one of my Hyper-V servers started rebooting all by itself. Oddly enough, the root cause of that mystery was an APC that hiccupped every so often (it took several months for me to be coincidently in the server room and next to the SU2200 when it happened). Oh well.

Actually, we are making progress! For the first time, I know understand that the bar graphs SHOULD NOT be showing up with the latest firmware 6.0.6. That is why I kept sending the screen shots to you .... they had NO meaningful data! (The bars were empty). As I said in my earlier email, as a novice, it seems to me that it is the embedded 9630 web page that has the error (I am getting the OLD bar graphs from the 9630, right???). To me, the code for the bar graphs must be buried somewhere in the new 9630 firmware, in error.

Of greater interest is . . . why do I only get the error (OLD Quick Start)  from one out of 4 work stations on the LAN ..... all using the same IE 10, Chrome and Firefox? Very weird.

I have done nothing to the SMT750/9630 between 9:43AM and the last e-mail. It sat all by itself on the bench connected to the LAN. No load, nothing. However, I think I am not being clear. Even before the format, sometimes I would get the OLD bar graphs and sometimes not  . . . . on my work station. I ALWAYS got the correct start page (no bar graphs) from the other three work stations. Also, I ONLY get the old bar graphs (on my work station) with IE 10 (I believe). Never, on any work station using Chrome or Firefox. Even weirder......

Maybe I don't understand the logs, but I don't think we are randomly losing communication. I think, what you are seeing is me restarting/booting the 9630 by pulling the card from its slot and or brain deading the SMT750. . . . all hoping to "fix" the issue (those darn bar graph vestiges showing up).

Have I confused you now too???????????? So, I think the real question is, why are the bar graphs showing up.? Where are they coming from? Why does it only happen with IE10?

How would you proceed with this issue?

Thanks!

Charlie

[BillP]

Ok, I am working on finding out about the bar graphs.

Just to confirm, currently, right this minute, you can see the correct page without the bar graphs, on every other system/browser BESIDES IE10 indicating communication is successfully established and the card looks and act normally everywhere besides IE10? And compatibility mode makes no difference?

I agree with you - that home.htm page is not working properly I just need to narrow down how to explain when you are seeing and when you are not. I initially thought it was showing incorrectly all of the time and only when the UPS had no communication but by reading what you're saying, that is not true.

That way, I can narrow this down and get it fixed for us.

[Anonymous user]

Ahhhh... I feel 6.06.07 coming on!

"Just to confirm, currently, right this minute, you can see the correct page without the bar graphs, on every other system/browser BESIDES IE10 indicating communication is successfully established and the card looks and act normally everywhere besides IE10? And compatibility mode makes no difference?" YES . Additionally, SOMETIMES the 9630 home page loads and displays correctly even in IE10 on my work station. Also, I have only been able to reproduce the vestige bar graph page on ONE work station, regardless of browser. I still think there is a problem with the new home page as served up by the 9630.

I have a second 9630 showing up in a couple of days, I will sub that into this SMT750 and see if it reacts differently. If it does, I will beg you for an RMA on the original (you can test the heck out of it and make it give up the errant code). If it doesn't, I will beg you for a software fix ........

Thanks for all the help.

Charlie

[Anonymous user]

Angela,

I hope you are not still working!!!

I think I figured it out........

Go to http://192.168.1.240/uphome.htm where the IP is YOUR IP for your NMC... but open page "uphome.htm".  This is the URL that I was going to on my work station. It was fine with the OLD firmware, but not the new! If you go there, I bet you see the same bar graphs too with your NMC!!!!!!

I was going to my APC/NMC using a "favorite" in IE 10. The URL references "uphome.htm rather than "home.htm". It seems that "uphome.htm" must still be buried in the new firmware!!!!!!!

That explains why I was only getting the bar graphs on ONE WORK STATION AND ONLY IN IE 10. My work station was the only one with that saved favorite . . .  using the URL, uphome.htm!!!!!!!!!!!!!

You all need to remove whatever is left of UPHOME.HTM from all future firmware updates!

Did I get this one right? Problem solved?

Charlie

[Anonymous user]

Confirmed!!  I have tested going to the URL "uphome.htm" rather than "home.htm" on all workstations and all browsers when opening an NMC "start" page". Yes, I can recreated the bar graph page without data everywhere. I hope you can too!

As long as the bar graphsshould be gone in the 6.0.6 firmware, crisis over, problem solved! I hope uphome.htm is completely deleted in the next iteration.

Thank you again Angela!

[voidstar_apc]

FYI, uphome.htm is used for older UPSLink UPSes. The bar graph probably still exists for those.

[BillP]

Ok here is what I know. And Charlie, yes you're essentially correct with your latest findings.

This page (uphome.htm) is only accessible by going to it directly - which makes sense since you had it bookmarked Charlie Ojserkis. My argument with you () is that I would not recommend you bookmark specific URLs because stuff like this can happen where we change web pages between versions. This page was removed and should never come up if you navigate to it directly and I will log a request to remove it completely in a future revision. I'd suggest that anyone reading this who may encounter it too stick to just bookmarking the IP or FQDN of the device to avoid the problem of stale web pages. That is what I normally do so that is why I think I have never encountered it.

So, while it is confusing to the users, I don't think it is CRITICAL that we remove it like right this second since it should not be popping up unless a user types it in directly like you effectively did. And obviously, we know the situation now.

Regardless, I've learned my lesson now with checking the URLs and we'll address this one in a future revision.

P.S. - voidstar - I checked a symmetra LX (UPS-Link) and it still goes to just home.htm. My understanding is the bar graphs were removed as part of the new web UI and should not come up for any type of UPS yet the page still exists if accessed directly. The only difference I saw on the symmetra LX was that the values actually populated properly where as for Charlie's, they were all blank and looked like a no comm with UPS page so that was the initial concern.

[voidstar_apc]

Ah, my bad.

[BillP]

I just double checked though for uphome.htm on a Micro-link on v5.1.7 and it does the same thing as 6.0.6 did for Charlie - so I think we're both right to an extent

So, I do not think any of this behavior changed for 6.0.6 and the only thing that changed is the type of UPS (micro-link vs. UPS-Link) changed here.

uphome.htm on a micro-link UPS:

If I use home.htm on this firmware, I see:

So, in v5.1.X, if you go to uphome.htm on a Micro-link UPS, you would see the same issue - the user is not supposed to need to worry about all of this when they go to log in though so another good reason not to bookmark anything beyond the IP or FQDN.

makes sense that this indeed for uphome.htm and only still works today if the UPS is UPS-Link.

[Anonymous user]

Angela,

A good learning experience all the way around. Without your help and support I/we would have never ever figured this out. THANK YOU.

As far as the URL favorites go, I agree with you, sort of. Managing 20 plus APC devices, you need the favorites to keep one's sanity. I never paid any attention to what got saved as the URL, because it always worked (until now) and that goes back 10 plus years. The only reason it is an issue now is because of sloppy programming (which I bet isn't done in RI ). In my case (and all cases), trying to load uphome.com should yield a page not found error, not some vestige of the once useful page. Obviously, this has never been an issue in the past, or you would have been familiar with the problem. No, it is not critical that it get removed immediately, but I would think the next iteration should have it gone. Just as an aside, I do miss my bar graphs  . . . . . .

Thanks again!

Charlie

[BillP]

No problem. Also, our programming for this particular application is done between RI and Mass

[Anonymous user · Accepted solution]

Ooops. I just assumed that since the cards appear to be manufactured in India, that they did the software there too.