Subscribing is a smart move!
You can subscribe to this board after you log in or create your free account.
Invite a Co-worker
Send a co-worker an invite to the portal.Just enter their email address and we'll connect them to register. After joining, they will belong to the same company.
Send Invite Cancel
Solved
Go to Solution
Accepted Solutions
Link copied. Please paste this link to share this article on your social media post.
Well it's been a couple of years! The second unit finally failed a couple of months back with the same symptom, and so I had another look at it and I've worked out what the problem is. Both units are now working again.
On the NMC the rear board (the one without the 7-segment display on it) has a 47uF capacitor on it. This capacitor seems to fail as it was only measuring around 15uF on both failed units (both in circuit and out of circuit). One of them also had a little electrolyte leaking out that I could only see as it reflected a tiny bit of light when I was wiggling it to desolder it.
Replacing this capacitor with another one fixed the problem on both units.
One of the units had tried to reflash the ATC baseboard firmware while it was in the process of failing (many months ago) which caused the repaired unit to still not function correctly (symptom: front LEDs flash in order from left to right, with LED marked "output" staying lit the whole time, unit reports "Lost communication with ATC" via serial console or web page). As discussed above, the solution was to get the NMC to reflash the baseboard firmware. It tried and failed to do this at powerup, however by holding down the Preference button on the front panel while powering on the unit, and not letting go of the button until the 7-segment display was showing "0" caused it to reflash the firmware (this is happening when the 7-seg display shows a "d" on the left digit and an animation on the right digit). It kept flicking back to "d0" briefly a number of times which I thought meant it was restarting the flash process but after about two minutes the relays clicked and everything was back to normal.
Both units are now back in service, so hopefully this will help anyone else who may encounter failed units one day!
Link copied. Please paste this link to share this article on your social media post.
Hmm, yes, not sure exactly why that is happening as you're saying and showing with that event log. There doesn't seem to be an obvious problem with that preference button sticking, right? I know you don't know the history so we are at a loss there if something that happened caused this odd behavior.
Not sure if you can do this (if you have attached load being powered) but what if you unplug the ATS from both sources and let it sit a minute (AKA a coldstart) and boot it back up?
What was in the event log from the when the unit appeared to recover after sitting the first time?
Another thing you can try at your own risk after removing all AC power from the ATS (and I can't recommend it officially since this product isn't user servicable but I know you opened it already anyway 
) but you can open the lid again and reseating the ATS controller card in its slot. It is the piece that looks like a stick of RAM. I'd pull that out and verify you reseat it properly and securely. Can check the behavior after that as well.
I agree the erratic cycling of the LEDs from left to right is indicating the firmware download is starting/failing.
Link copied. Please paste this link to share this article on your social media post.
The preference button seems fine - I can use it to change preferences without any trouble and the preference LEDs don't change on their own once set, so doesn't look like there's anything amiss with the button itself.
You're right, I have opened the unit 
I did try reseating the "RAM stick" ATC controller originally (didn't realise it was the whole controller, just assumed it was the firmware flash chip) but it didn't change anything. There's no load as these units are out of service and just on my desk for testing. They've had the power off overnight for a few nights running so they have been cold started every morning. For some reason once the controller gets into firmware flash mode, it doesn't seem to exit it (not even after a coldstart) and the only way to get out of that mode seems to be to provide the data to flash.
So because of this, the unit did not come back after this latest test of holding down the prefs button, having gotten stuck in flash-mode. I actually have two of these units, the other one appearing to work fine, so as a test I swapped the network controllers (NMC) so that this suspect unit has a known good NMC. Since this gave me a working LED display, I could see it briefly show d' as it started to download the firmware but almost instantly it flicked back to 0 as the download failed. I left it a while but it did not successfully update the firmware, nor did it really seem to try again, even though the controller was stuck in firmware flash mode. I then realised the NMC firmware was older than the original NMC (3.7.2 vs 3.9.2), so I reflashed it via FTP to the latest version. After the second g2ats firmware file was uploaded and the NMC rebooted, it did try to reflash the controller firmware and for some reason this time it succeeded on the first attempt, and the unit returned to a working state. This time I saw the LED display show a "d" and an animation during reflashing, and the other LEDs cycled quickly left-to-right but at a uniform speed.
I'm a bit hesitant to try the prefs button again (to see whether this was a bug that's been fixed in the latest firmware version) seeing as the unit appears to work fine now (apart from the dead 7-seg display). But if there's anything you'd like me to try to satisfy your own curiosity, let me know. At any rate, the solution for a G2ATS unit stuck in "cycling LEDs" mode seems to be to just reflash the NMC firmware and then leave it for a few minutes afterwards.
As far as what was in the logs after the first time it recovered, it's difficult to say because I'm not sure exactly what the time was, and the older log messages are lost because of all the download begun/fail/begun/fail messages. I think this is the relevant section, taken from the longer log file you can get via FTP:
08/15/2018 17:46:51 System: Console user 'apc' logged in from serial port. 0x0014 08/15/2018 17:46:24 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:46:23 Automatic Transfer Switch: ATS Controller firmware download completed. 0x0C1D 08/15/2018 17:45:20 Automatic Transfer Switch: 7-Segment display hardware failure. 0x0C20 08/15/2018 17:45:20 Automatic Transfer Switch: Phase Leds/Push-button hardware failure. 0x0C20 08/15/2018 17:44:59 System: Warmstart. 0x0002 08/15/2018 17:44:23 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:23 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:22 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:22 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:21 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:21 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:20 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:19 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:18 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:18 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:17 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:17 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:16 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:16 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:15 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:15 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:44:14 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:44:14 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:42:48 Automatic Transfer Switch: 7-Segment display hardware failure. 0x0C20 08/15/2018 17:42:48 Automatic Transfer Switch: Phase Leds/Push-button hardware failure. 0x0C20 08/15/2018 17:42:27 System: Warmstart. 0x0002 08/15/2018 17:41:09 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:41:08 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:41:08 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:41:07 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E 08/15/2018 17:41:06 Automatic Transfer Switch: ATS Controller firmware download begun. 0x0C1C 08/15/2018 17:41:05 Automatic Transfer Switch: ATS Controller firmware download failed. 0x0C1E
What I don't really understand is why it completed when there was no 'download begun' message that came first. And why did it keep trying to download the firmware and keep failing, once the unit was working again and comms had been established? Looking further at the logs it seems that the unit kept trying to download the firmware, even after it was successful, until it was next powercycled. When it had a coldstart event with working firmware there were no further download attempts until I held the prefs button as mentioned before. Looking now at the latest logs after the unit was off overnight after a successful flash, there are no further messages at all referring to firmware downloads.
Thanks again Angela for your help with this. I wish we had access to all the internal documentation you do because it would be fascinating to read! I did ask APC Australia and they said they have no problem sharing the stuff (service manuals etc.) for all their discontinued products, but they don't have copies of it as only the repair places do, and sadly I wasn't able to contact anyone via e-mail who had access to it. If you do, I'd love to hear from you!
Here's a quick photo for your amusement. Yes that's not the real APC serial cable, it's a Cisco one that I put an RJ12 connector on, real professional stuff
Link copied. Please paste this link to share this article on your social media post.
Hmm, I am not quite sure what is going on there after reading what you explained.
These products are "replace only" worldwide as far as I am aware so there is no secret technical service manual that I or anyone has unfortunately 
. Anything that may be out there wouldn't be authorized or provided by us for the reason I just mentioned as we don't provide schematics or anything else.
On 8/16/2018 10:42 PM, Adam said:For some reason once the controller gets into firmware flash mode, it doesn't seem to exit it (not even after a coldstart) and the only way to get out of that mode seems to be to provide the data to flash.
That is interesting. I've never seen a unit get stuck in a mode like this. I wonder what happened to this unit prior to cause this type of behavior. I have to assume something has been damaged electrically here?
Link copied. Please paste this link to share this article on your social media post.
Hi Adam,
On 8/25/2018 5:53 AM, Adam said:Thanks again for your help with this, I really appreciate it!
I don't know how helpful I am but thanks 
On 8/25/2018 5:53 AM, Adam said:Re the service manuals and schematics, do you know who within APC/Schneider who would have access to these? The Australian APC folks weren't sure who to go through, so I'm thinking if I can point them in the right direction then maybe that would help. Of course all this stuff would be unofficial and not supported, but it sure helps people like me if we can get access to schematics and things, especially when we're working with long discontinued products like the old Smart UPS units.
While I definitely hear and understand what you're saying and the reasons why you're looking for this type of info, we don't and can't provide schematics for anything as they contain proprietary information. There is also a safety concern with releasing this information as the PDU and ATS products just aren't designed to be user serviceable. Just our engineering teams have access to the level of information you're requesting. There are no secret/internal service guides that anyone in tech support or any other dept has access to for these offers since they don't exist. Whatever exists is just what engineering/design teams develop and document which is not shared outside of their group per what I just explained.
We have an FAQ on this here and while it focuses on UPS, the general statement applies to PDU/ATS and most of our products. (I am not super familiar with 3ph and cooling solutions but I imagine it may be the same.) http://www.apc.com/us/en/faqs/FA158607
Link copied. Please paste this link to share this article on your social media post.
Well it's been a couple of years! The second unit finally failed a couple of months back with the same symptom, and so I had another look at it and I've worked out what the problem is. Both units are now working again.
On the NMC the rear board (the one without the 7-segment display on it) has a 47uF capacitor on it. This capacitor seems to fail as it was only measuring around 15uF on both failed units (both in circuit and out of circuit). One of them also had a little electrolyte leaking out that I could only see as it reflected a tiny bit of light when I was wiggling it to desolder it.
Replacing this capacitor with another one fixed the problem on both units.
One of the units had tried to reflash the ATC baseboard firmware while it was in the process of failing (many months ago) which caused the repaired unit to still not function correctly (symptom: front LEDs flash in order from left to right, with LED marked "output" staying lit the whole time, unit reports "Lost communication with ATC" via serial console or web page). As discussed above, the solution was to get the NMC to reflash the baseboard firmware. It tried and failed to do this at powerup, however by holding down the Preference button on the front panel while powering on the unit, and not letting go of the button until the 7-segment display was showing "0" caused it to reflash the firmware (this is happening when the 7-seg display shows a "d" on the left digit and an animation on the right digit). It kept flicking back to "d0" briefly a number of times which I thought meant it was restarting the flash process but after about two minutes the relays clicked and everything was back to normal.
Both units are now back in service, so hopefully this will help anyone else who may encounter failed units one day!
Create your free account or log in to subscribe to the board - and gain access to more than 10,000+ support articles along with insights from experts and peers.