AP7930 firmware won't flash

[Anonymous user]

Hi, I'm hoping someone here can help. I'm trying to re-flash firmware to resolve a network issue. The PDU will not reset the network preferences. I tried resetting the entire unit, just the network... changed it from static to bootp.. it always keeps the old values.

So I tried to flash it, and it fails. Below is the log.

Log:

Using AOS file        : AOS = apc_hw02_aos_374.bin

Using Application file: APP = apc_hw02_rpdu_374.bin

Checking network connection ...  Checking version information ...

  *************** Next **************

Before Upgrading the NMC the AOS is: apc_hw02_aos_373.bin

Attempting to log in to 10.0.31.35 ... Log in sucessful

Loading OS, please wait ...                      OK

Please wait (2 minutes ) for system restart   OK

Attempting connection to verify restart        OK

Attempting to log in to 10.0.31.35 ...         OK

Loading application, please wait ...             OK

Please wait (30 seconds) for system restart   OK

Attempting connection to verify restart        OK

After the upgrade process has completed,

looking for AOS: apc_hw02_aos_374.bin

Found AOS      : apc_hw02_aos_373.bin

looking for APP: apc_hw02_rpdu_374.bin

Found APP      : apc_hw02_rpdu_374.bin

All upgrades completed successfully.

However the CMD window (can't copy from it) gives "Verifying 10.0.31.35 upgrade ... Error, see log for details. The log only shows success.

What I noticed is that the APC never reboots during the process, I can always ping it. I can't manually reboot it since the upgrade tool is logged in, and it's 1 login max at all times.

Any ideas/tips to get this working again?

Thanks!

[Anonymous user]

Hi, I'm hoping someone here can help. I'm trying to re-flash firmware to resolve a network issue. The PDU will not reset the network preferences. I tried resetting the entire unit, just the network... changed it from static to bootp.. it always keeps the old values.

So I tried to flash it, and it fails. Below is the log.

Log:

Using AOS file        : AOS = apc_hw02_aos_374.bin

Using Application file: APP = apc_hw02_rpdu_374.bin

Checking network connection ...  Checking version information ...

  *************** Next **************

Before Upgrading the NMC the AOS is: apc_hw02_aos_373.bin

Attempting to log in to 10.0.31.35 ... Log in sucessful

Loading OS, please wait ...                      OK

Please wait (2 minutes ) for system restart   OK

Attempting connection to verify restart        OK

Attempting to log in to 10.0.31.35 ...         OK

Loading application, please wait ...             OK

Please wait (30 seconds) for system restart   OK

Attempting connection to verify restart        OK

After the upgrade process has completed,

looking for AOS: apc_hw02_aos_374.bin

Found AOS      : apc_hw02_aos_373.bin

looking for APP: apc_hw02_rpdu_374.bin

Found APP      : apc_hw02_rpdu_374.bin

All upgrades completed successfully.

However the CMD window (can't copy from it) gives "Verifying 10.0.31.35 upgrade ... Error, see log for details. The log only shows success.

What I noticed is that the APC never reboots during the process, I can always ping it. I can't manually reboot it since the upgrade tool is logged in, and it's 1 login max at all times.

Any ideas/tips to get this working again?

Thanks!

[BillP]

keep in mind after any resetting of credentials, you'll need to log off for them to take effect which can take up to 30 seconds after the log off.

Anyway, here is what I would do. Are you familar with FTP? We'll use the FTP instructions listed in article ID FA156047 in our knowledge base (http://www.apc.com/site/support/index.cfm/faq/index.cfm ) and refer to manual FTP for NMC1.

What I need you to do is access the PDU via FTP and tell me what you see in bold for your version number please seen below:

220 AP7941 Network Management Card AOS v3.7.3 FTP server ready.

User (10.218.44.131:(none)):

Now, can you log in via FTP with your admin username and password and do the ls command and tell me what .bin files are there. I suspect you will have AOS 3.7.3 and you have rpdu 3.7.4.

If that is correct, we need to FTP rpdu 3.7.3 bin to the card so that it will work with AOS 3.7.3 and then we can reconnect to the PDU and FTP AOS 3.7.4 over, let it reboot, then repeat with rpdu 3.7.4 and it should be fine.

Basically, you just have some mismatched files on there so far so they don't work properly together.

So, review the manual FTP instructions, verify what files are on the PDU now, and I am correct that you see AOS 3.7.3 bin and 3.7.4 rpdu application, then proceed with:

1. Put rpdu 3.7.3 on the card (PDU) then log out per the FTP instructions in the kbase (to allow for reboot after the file).
2. Put AOS 3.7.4 on the card (PDU) then log out per the FTP instructions in the kbase (to allow for reboot after the file).
3. Put rpdu 3.7.4 on the card (PDU) then log out per the FTP instructions in the kbase (to allow for reboot after the file).

If am I wrong with any assumptions, let me know before you start.

[Anonymous user]

Hi Angela,

thanks for the reply. The problem is that it won't stay online more than 5-6 seconds. After that I can't ping or connect to it for 2-3 minutes, then it's up again for another 5-6 pings. So logging in by FTP is impossible.

However I noticed that this problem went away during the flash. I'm not sure what it does, but if I start the flash process while it pings, then it will keep pinging until the process is done, even if it fails.

Can I downgrade to 3.7.3 from the flash utility or will that fail also?

Thanks again for your help!

[Anonymous user]

No I've been looking for it and can't find it. Are you guys able to provide it? Or is there an archive FTP where those older files reside?

Thanks!

[BillP]

You can try it! Do you have the 3.7.3 file?

[BillP]

I will attach it here for you.

[Anonymous user]

Still no go

I first downgraded to 3.7.3 first, then it went dead. I hit the reset button, tried to edit settings... still no go. But when I launched the 3.7.4 update, I saw it reboot (ping stopped for a little while then resumed). It loaded the application, restarted again (confirmed again with ping).

After the update, it shows the proper firmware versions:

American Power Conversion	Network Management Card AOS	v3.7.4
(c) Copyright 2009 All Rights Reserved  Rack PDU APP	v3.7.4

After the process was completed (without any errors this time), it stopped pinging. The issue resumed as before, with 5-6 pings then 2-3 minutes dead. I changed the network cable and the switch port, to no avail.

At this point I'm guessing it's a problem inside the unit. We'll plan a replacement and RMA it.

Thanks for your help Angela, really appreciated!

[BillP]

You're welcome. If you still have time to look at it, I'd be interested in the visual indicators you see on it when it pings and when it does not ping. I feel like we should be able to fix an issue like this (with what I know so far) but if you're done playing, I understand.

[Anonymous user]

Since it's live I don't mind playing with it a little more, it'll be a hassle to replace it anyway. You know how it is

The visual indicators are as follows:

B1 = orange (bright)

B2 = amber (a little dimmer)

Amps = 8 (even if I press the bank button)

The NIC green lights is always on, and the NIC amber lights is flickering with activity.

This is the same if it pings or not, those are static.

[Anonymous user]

Just so you can see, I made a timestamp ping script:

# ping -A 10.0.31.35 | while read pong; do echo "$(date): $pong"; done

Mon May 13 15:03:01 EDT 2013: PING 10.0.31.35 (10.0.31.35): 56 data bytes

Mon May 13 15:03:01 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=60 ttl=252 time=1167.876 ms

Mon May 13 15:03:02 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=62 ttl=252 time=3.775 ms

Mon May 13 15:03:03 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=63 ttl=252 time=3.044 ms

Mon May 13 15:03:04 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=64 ttl=252 time=3.675 ms

Mon May 13 15:03:05 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=65 ttl=252 time=2.840 ms

Mon May 13 15:03:06 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=66 ttl=252 time=6.021 ms

Mon May 13 15:04:33 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=151 ttl=252 time=1479.153 ms

Mon May 13 15:04:34 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=153 ttl=252 time=3.083 ms

Mon May 13 15:04:35 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=154 ttl=252 time=3.665 ms

Mon May 13 15:04:36 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=155 ttl=252 time=17.322 ms

Mon May 13 15:04:37 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=156 ttl=252 time=3.479 ms

Mon May 13 15:04:38 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=157 ttl=252 time=4.026 ms

Mon May 13 15:06:05 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=242 ttl=252 time=1575.670 ms

Mon May 13 15:06:05 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=243 ttl=252 time=569.610 ms

Mon May 13 15:06:06 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=244 ttl=252 time=2.592 ms

Mon May 13 15:06:07 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=245 ttl=252 time=3.151 ms

Mon May 13 15:06:08 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=246 ttl=252 time=5.028 ms

Mon May 13 15:06:09 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=247 ttl=252 time=2.958 ms

Mon May 13 15:06:10 EDT 2013: 64 bytes from 10.0.31.35: icmp_seq=248 ttl=252 time=3.530 ms

[BillP]

First - how did this problem come about or led to the discovery of it?

Do you think it is possible the jack connection is loose on the PDU end? Is it possible to try a direct connection to a computer to see if the ping results are still resulting in lots of packet loss?

I was sort of expecting the PDU to be rebooting based on what you've said but if the LED is not changing (the one with the checkmark or X next to it called the status LED). Anything interesting in the log recently? i.e. lots of warmstart events or similar? I'd be interested in seeing that or at least what you can see if you can sneak in to view it. Don't know if you can via serial cable either since the network is flaky..

Also, did you add more to your post maybe after I read it the first time or am I just really bad at reading? The first time I looked I did not see the part about how it is actually at 3.7.4 now but now I do.

The other thing that came to mind was duplicate IP but I feel like the behavior might be a little different.

[Anonymous user]

The problem appeared out of nowhere honestly. We monitor all our PDUs with SNMP, and I started seeing it going up and down last week. We also have icmp monitoring which reported the ping timeout issues.

I tried with my laptop and the same results. If I check the uptime using the serial console, I see "Up Time   : 0 Days 0 Hours 0 Minutes" so it looks like it's really rebooting in a loop. Both power supply status are reported OK.

I have to leave the office for the day now but I'll get back on it tomorrow morning and let you know what I can see in the logs. Serial appears to be working provided I reset it before. That allows me to stay a full minute in before it kicks me out again for another reboot!

Did you mean post #6? I don't think I edited it to add the firmware, but hey long day! After the downgrade and upgrade, it's all at 3.7.4 now!

[BillP]

Yup - post 6 I must have missed something there initially.

But anyway, OK, so the PDU does appear to reboot according to the uptime but you're not seeing the LEDs reflect it (or its quick and we are missing it somehow). The status LED would alternate green and orange.

If you are able to serial in, can you do Ctrl-L to show the event log and at least report the gist of events you see?

I am wondering if while you're in there for a minute or after glancing at the event log and dealing with a few "kick-outs," you'd like to try and reset the entire unit to defaults? This is under System menu->Tools->Reset to defaults.

It will wipe everything including TCP/IP settings so it'd require a complete reconfiguration (if it even works). I believe when you do this command, you'll have to type yes to confirm it then hurry up and Esc and press log out for it work. I don't have one I can reset now or else I'd verify those last few steps.

[Anonymous user]

Ok you're right, the status light does start blinking when it reboots. I tried resetting to defaults, resetting only TCP/IP, and everything but TCP/IP. It still keeps the network information there.

Here's an excerpt from the log:

	Date	Time	Event
	--------------------------------------------------------------------------
	05/14/2013  03:24:33  System: Console user 'apc' logged in from serial
	port.
	05/14/2013  03:24:33  System: Network service started. System IP is
	10.0.31.35 from manually configured settings.
	05/14/2013  03:24:33  System: Warmstart.
	05/14/2013  03:22:54  Switched Rack PDU: Bank configuration change.
	Overload threshold on bank 1.
	05/14/2013  03:22:54  Switched Rack PDU: Bank configuration change. Near
	overload threshold on bank 1.
	05/14/2013  03:22:54  System: Initializing data in the DCal file.
	05/14/2013  03:22:54  System: Initializing data in the PDU Action file.
	05/14/2013  03:22:54  System: Initializing data in the Syslog file.

	05/14/2013  03:24:36  System: Configuration change.  Syslog server port  1
	port.
	05/14/2013  03:24:36  System: Configuration change.  Syslog server IP  1
	address.
	05/14/2013  03:24:36  System: Configuration change.
	05/14/2013  03:24:36  System: Initializing data in the
	EventLogConfiguration file.
	05/14/2013  03:24:33  System: Console user 'apc' logged in from serial
	port.
	05/14/2013  03:24:33  System: Network service started. System IP is
	10.0.31.35 from manually configured settings.
	05/14/2013  03:24:33  System: Warmstart.

	05/14/2013  03:24:36  Switched Rack PDU: Bank configuration change. Near
	overload threshold on bank 2.
	05/14/2013  03:24:36  Switched Rack PDU: Bank configuration change.
	Overload threshold on bank 1.
	05/14/2013  03:24:36  Switched Rack PDU: Bank configuration change. Near
	overload threshold on bank 1.
	05/14/2013  03:24:36  System: Initializing data in the DCal file.
	05/14/2013  03:24:36  System: Initializing data in the PDU Action file.
	05/14/2013  03:24:36  System: Initializing data in the Syslog file.
	05/14/2013  03:24:36  System: Configuration change. Syslog, re-mapped local
	severity: 'None'.
	05/14/2013  03:24:36  System: Configuration change. Syslog, re-mapped local
	severity: 'Informational'.

The rest is mostly stuff like:

	05/14/2013  03:24:36  Switched Rack PDU: Outlet configuration change.
	Outlet Link on outlet #3 (Outlet 3).

This is after I tried resetting to defaults (everything):

	Date	Time	Event
	--------------------------------------------------------------------------
	05/14/2013  03:28:54  System: Initializing data in the
	EventLogConfiguration file.
	05/14/2013  03:28:54  System: Initializing data in the Event Action file.
	05/14/2013  03:28:49  System: Network service started. System IP is
	10.0.31.35 from manually configured settings.
	05/14/2013  03:28:52  System: Configuration change. Secondary RADIUS server
	address.
	05/14/2013  03:28:52  System: Configuration change. Primary RADIUS server
	address.
	05/14/2013  03:28:52  System: Initializing data in the DNS file.
	05/14/2013  03:28:52  System: Configuration change. NTP update interval.
	05/14/2013  03:28:52  System: Configuration change. NTP GMT offset.

As you can see, it still reads the IP from configured settings even after a full reset. Next I try just a network reset:

	05/14/2013  03:34:09  System: Initializing data in the Event Action file.
	05/14/2013  03:34:06  System: Console user 'apc' logged in from serial
	port.
	05/14/2013  03:34:06  System: Network service started. System IP is
	10.0.31.35 from manually configured settings.
	05/14/2013  03:34:06  System: Warmstart.

Hopefully this gives you a better idea of the problem!

[BillP]

Hm, yes, sounds like its not resetting. When you try the resets of either type, does it ask you to type yes to confirm and log out as I mentioned previously?

[Anonymous user]

It asks me to confirm, iI type "YES" then depending on the type of reset things change.

If I do just a network reset, changes will take effect at logout. So I logout, but nothing changes even after waiting for 2-3 minutes.

If I do a complete reset, it terminates the session as soon as I hit enter after "YES" so I never have time to logout.

It's connected through serial on an IP KVM now, I'll be able to test from anywhere as long as I don't have to press the reset button!

[Anonymous user]

Here's a full reset:

	-----------------------------------------------------------------------
	Reset to Defaults

Enter 'YES' to continue or to cancel : yes

	Now initializing system to default values including
	all User Names, Passwords and TCP/IP settings.
	Please wait...

Then I have to wait 2-3 minutes for it to come back up, I never get a chance to logout.

[Anonymous user]

Yup, exactly. If I keep hitting enter eventually I get to a login prompt.

[BillP]

OK - I am stumped. I have not really seen one that refused to reset before. Have you power cycled it since this started (which I imagine is not something you can do right away)?

[Anonymous user]

Yeah it's hooked up live in the datacenter with all outlets used, so it's not something I can do without first replacing the unit.

Do you know of a way to manually delete the saved network information?

[BillP]

Not beyond resetting it to defaults. The other option could be to go into the network menu->TCP/IP and changing everything to zeroes and then changing it to DHCP mode? Cause if you switch DHCP mode it will still keep the manual settings, just make them inactive.

[BillP]

the next thing would be power cycle for me which i know we already discussed.

our newer stuff allows for a format of the file system by holding down the reset button but this one does not unfortunately or else we could do that

[Anonymous user]

Alright, thanks a lot for all your help, it's really appreciated. Once I have replaced the live unit with a new one, I'll update this post (though it may be some time).

[BillP]

No problem. Is it under warranty? Even if its not, I sort of want to get it back and look at it and see if someone can tell me what has happened..we could discuss that offline in the future how to make those arrangements.

[Anonymous user]

No I don't think it's still under warranty, we've had it for a while now. Once it's disconnected I'll PM you for arrangements if you still want it.

My boss suggested we try a serial firmware transfer, but I can't find instructions for this model. Would you be able to provide such instructions? I know other models have them online (96xx), but I only found hyper-terminal instructions, not putty/console instructions (I'm on a Mac). I'm familiar enough with the procedure through XMODEM on Cisco switches, this shouldn't be much harder, however it never prompts me for a file to upload or a location locally where it should be - nor can I pass options like in Cisco's OS.

[BillP]

PuTTy does not support XMODEM protocol for file transfer unfortunately. TeraTerm might as that is available for Mac I believe.

You'd follow the same instructions for XMODEM transfer as for NMC1 (AP9617/18/19) as listed in the knowledge base.

I personally think this will take a long time to complete and not do anything different since the application is not corrupted currently but it does not hurt to try.

We typically only do that if the network is unavailable or there is a problem FTPing.

[BillP · Accepted solution]

So pressing enter at the end of that please wait does nothing because it does reboot immediately?