AP7930 web interface - Error: The application was not able to load.

[cuddylier_apc]

Hi

My AP7930 has suddenly started showing the following when I try to access the web interface:

Error: The application was not able to load.

You are attempting to access an APC device. There was a problem loading the application. Please login to the device via telnet for more details.

When I login to the PDU via console, this is what it shows:

Stat : P+ N+ A-

I tried connecting to the FTP server but get the following errors:

Windows command prompt FTP client:

150 File status okay; about to open data connection.

Aborting any active data connections...

425 NLST can't open data connection.

Filezilla:

Response: 421 Service not available, closing control connection.

Above is while trying all transfer modes including passive, I presume it's not working partially due to the OS seemingly being messed up.

Any ideas on what I can do to bring the web interface back to life? SSH and Telnet are both dead, despite me enabling each (one at a time).

Thanks

[voidstar_apc]

I'd try the reset button (or remove the power if there's no reset button). If it doesn't work after that, the "format" command will blow away the filesystem (but leave AOS intact). Then configure networking settings. Check if ftp works. If so, grab the latest firmware for your device from APC.com and install it. If not, try eliminating the possibility of a network problem by plugging the AP9630 and your PC into another hub or switch.

[BillP]

Just one clarification  voidstar  - this is an AP7930 Rack PDU. I think you had it confused with AP9630 NMC2 so I wanted to comment to clarify for original poster. Too, this means there will be no format option or ability to move it between UPSs.

To the original poster, do you know for sure your FTP data port is open? Typically in this situation, FTP should still be working. The errors you were seeing I've seen if the FTP data port (20) is blocked. It is feasible that it is not working properly due to something else as you suggested but I'd also want to confirm about the FTP data port.

In this situation, we need to figure out how to re-load the application file. Through the serial connection or FTP prompt, do you know which AOS is on the PDU right now? We'll want to load the matching application to see if we can restore the PDU back to the correct state. Worst case, we could try it over a serial connection but hopefully we can get FTP working.

A power cycle (if possible - understand the PDU may have devices plugged in) or a press and release of the pinhole reset button may help so you can try those too. But, we may want to wait and proceed with caution because of the current state of the PDU and not 100% knowing how it got here or if there are other alarms/problems. In rare situations, if there are other problems, rebooting the PDU may cause the outlets to not re-energize after the reboot.

[cuddylier_apc]

On 12/4/2017 5:47 PM, Angela said:

Just one clarification  voidstar  - this is an AP7930 Rack PDU. I think you had it confused with AP9630 NMC2 so I wanted to comment to clarify for original poster. Too, this means there will be no format option or ability to move it between UPSs.

To the original poster, do you know for sure your FTP data port is open? Typically in this situation, FTP should still be working. The errors you were seeing I've seen if the FTP data port (20) is blocked. It is feasible that it is not working properly due to something else as you suggested but I'd also want to confirm about the FTP data port.

In this situation, we need to figure out how to re-load the application file. Through the serial connection or FTP prompt, do you know which AOS is on the PDU right now? We'll want to load the matching application to see if we can restore the PDU back to the correct state. Worst case, we could try it over a serial connection but hopefully we can get FTP working.

A power cycle (if possible - understand the PDU may have devices plugged in) or a press and release of the pinhole reset button may help so you can try those too. But, we may want to wait and proceed with caution because of the current state of the PDU and not 100% knowing how it got here or if there are other alarms/problems. In rare situations, if there are other problems, rebooting the PDU may cause the outlets to not re-energize after the reboot.

Hi Angelaand  voidstarthanks for replying!

I just checked again and it seems the FTP works now strangely.

I see the file apc_hw02_aos_392.bin in the root FTP directory.

Model Number      : AP7930

Serial Number     : ZA0732xxxxxx

Manufacture Date  : 08/10/2007

Hardware Revision : B2

MAC Address       : 00 C0 B7 72 xx xx

 Flash Type        : AMD A29DL322DB

 Name        : aos               Type        : APC OS

 Version     : 392               Sector      : 16

 Date        : 07/06/2016        Time        : 15:12:05

 CRC16       : C3FF

The PDU is in production so can't be powered off, I can try the reset pin though if you think it may work?

[cuddylier_apc]

On 12/4/2017 6:14 PM, Angela said:

I would try loading the 3.9.2 application file to the PDU first. I attached it to this post. Only problem is, the forum blocked it so I renamed it to .jpg. You will want to fix the attachment to be a .bin extension 

You'd want to FTP in, then do 'bin' and 'hash' commands to force binary mode and print hash marks, then 'put' this file (attached). Alternatively, I am sure you can find those same settings with FileZilla if not using a Windows command prompt  - Make sure to disconnect the FTP session when done and the management interface in the PDU will know to reboot after the file is done loading. Then, check afterwards and see what you get. If you get in, I suggest checking the event log to see if there is any clue to what occurred. The event log is only available when the app is correctly loaded.

If you need more details, look at NMC1 FTP instructions here -> http://www.apc.com/us/en/faqs/FA156047, specific to when it talks about loading the application file. (We don't need to re-load AOS which is the first few steps.)

I loaded the application file and that did the job, thank you!

I checked the event logs and surprisingly there wasn't anything at all referencing an issue with the OS, I do have a lot of near overload errors spamming the logs though so maybe they spammed so much that any other errors got hidden/ran off the end of the log?

[BillP]

There wouldn't be any messages specifically saying anything with the OS. I'd be looking for frequent "System: Warmstart" or TCP/IP stack failures or something like that. 

Feel free to attach a .txt file of the event log from FTP - you can retrieve event.txt using these instructions (if you need them). http://www.apc.com/us/en/faqs/FA156131 (copy/paste from web is OK too as long as you're showing the full log and I can see it from top to bottom, .txt is nicer format and less work if you can grab it.)

If you want, also remove any private info before posting the log. I'll take a look and see if there is anything in there that looks interesting to see what happened so we can also feel a little better it won't repeat itself.

[cuddylier_apc]

On 12/4/2017 7:44 PM, Angela said:

There wouldn't be any messages specifically saying anything with the OS. I'd be looking for frequent "System: Warmstart" or TCP/IP stack failures or something like that. 

Feel free to attach a .txt file of the event log from FTP - you can retrieve event.txt using these instructions (if you need them). http://www.apc.com/us/en/faqs/FA156131 (copy/paste from web is OK too as long as you're showing the full log and I can see it from top to bottom, .txt is nicer format and less work if you can grab it.)

If you want, also remove any private info before posting the log. I'll take a look and see if there is anything in there that looks interesting to see what happened so we can also feel a little better it won't repeat itself.

I do see warmstart in the event log as it came back to life. I've attached the event.txt

[BillP]

Thanks for sharing. That warmstart (soft reboot) is expected so we are OK there.

Is the 12/31/2016 date accurate, meaning was that last year or was that really a day or two ago or what?

I ask because I see this in the log 

12/31/201618:45:03System: File transfer failed.0x000F

But then I also see these, which if the date/time was wrong, I would've expected to see the correct date and time only occur as of these messages where it said it set the time (and newer messages), not way back with the System: Warmstart:

12/04/201719:31:01System: Set Date.0x002112/04/201719:31:01System: Set Time.0x0021

I ask because that File Transfer failed.. well if that was you today or the other day when we were fixing this OK, but if it was someone who attempted a firmware upgrade and it failed, it could've caused this problem.

I see an FTP log in a few minutes before that file transfer failed message..

12/31/201618:43:46System: FTP user 'apc' logged in from 10.10.x.x.0x0016

[cuddylier_apc]

On 12/4/2017 8:16 PM, Angela said:

Thanks for sharing. That warmstart (soft reboot) is expected so we are OK there.

Is the 12/31/2016 date accurate, meaning was that last year or was that really a day or two ago or what?

I ask because I see this in the log 

12/31/2016	18:45:03	System: File transfer failed.	0x000F

But then I also see these, which if the date/time was wrong, I would've expected to see the correct date and time only occur as of these messages where it said it set the time (and newer messages), not way back with the System: Warmstart:

12/04/2017	19:31:01	System: Set Date.	0x0021
12/04/2017	19:31:01	System: Set Time.	0x0021

I ask because that File Transfer failed.. well if that was you today or the other day when we were fixing this OK, but if it was someone who attempted a firmware upgrade and it failed, it could've caused this problem.

I see an FTP log in a few minutes before that file transfer failed message..

12/31/2016	18:43:46	System: FTP user 'apc' logged in from 10.10.x.x.	0x0016

That date/time was probably from just before I successfully got the new OS uploaded as that date seems to be right before the accurate date/time.

[BillP]

Hmm, OK. So you think when it set the date/time, it truly only had to update the time then? Or did I mis-understand? I think I misunderstood because in your original post, you made it seem like you log into the PDU pretty regularly and there is no way this problem existed for a year.

The only thing I see in the log in general is that failed file transfer. Not seeing anything else helpful.

Basically, when you upload a new .bin file, whether a new AOS or application file, it overwrites the old application file. If that file you chose was wrong or invalid and doesn't pass some checks, the Network Management Card will not transfer it to flash and thus, the app file space is left empty.

What I was suggesting as a possibility was that if someone tried to start a firmware upgrade and it failed for one reason or another, this could've occurred where the app file was wiped as it can't be recovered after a failed transfer. Other than that, not sure what happened based on the log..

[cuddylier_apc]

On 12/4/2017 8:32 PM, Angela said:

Hmm, OK. So you think when it set the date/time, it truly only had to update the time then? Or did I mis-understand? I think I misunderstood because in your original post, you made it seem like you log into the PDU pretty regularly and there is no way this problem existed for a year.

The only thing I see in the log in general is that failed file transfer. Not seeing anything else helpful.

Basically, when you upload a new .bin file, whether a new AOS or application file, it overwrites the old application file. If that file you chose was wrong or invalid and doesn't pass some checks, the Network Management Card will not transfer it to flash and thus, the app file space is left empty.

What I was suggesting as a possibility was that if someone tried to start a firmware upgrade and it failed for one reason or another, this could've occurred where the app file was wiped as it can't be recovered after a failed transfer. Other than that, not sure what happened based on the log..

I should have clarified, the previous date/time is wrong so it wasn't really a year ago, I do in deed login relatively regularly, every month or two. I'll just assume it was some sort of firmware issue and hopefully won't see it again.

Thank you once again for all your help, much appreciated.

[BillP · Accepted solution]

Ok, got you. No problem! Glad it is all set now.