SMX1500RM2U switching to battery

[Anonymous user]

We have a brand new SMX1500RM2U with a AP9630 V2 card that we have installed in the last couple of weeks. Since installation we have been seeing random switches to battery with the following warning:

Date : 03/30/2010
Time : 20:05:11
Code : 0x0109

Warning Events - UPS: On battery power in response to unknown.


Within 1 second it switches back to normal power with this warning:

Date : 03/30/2010
Time : 20:05:12
Code : 0x010A

Informational Events - UPS: No longer on battery power.

We have other equipment installed in this closet on the same circuit but NOT connected to the UPS and it isn't indicating that the AC power has been lost. We have both the most current SUMX code and UPS OS code of 5.11 loaded.
I have searched through the forums and haven't seen anything related to this. However, this is fairly new firmware. are there any known issues out there that would be causing this? At this point it doesn't appear to be facility power as mentioned above. It does appear to be a problem with the UPS.
I should also mention that we have ran both self tests and calibrations without issues.
Any help would be greatly appreciated.

[Anonymous user]

Yep, will do.

Thanks!

[BillP]

thanks. sent you a follow up email requesting some info so that we can take this offline.

[Anonymous user]

We have a brand new SMX1500RM2U with a AP9630 V2 card that we have installed in the last couple of weeks. Since installation we have been seeing random switches to battery with the following warning:

Date : 03/30/2010
Time : 20:05:11
Code : 0x0109

Warning Events - UPS: On battery power in response to unknown.


Within 1 second it switches back to normal power with this warning:

Date : 03/30/2010
Time : 20:05:12
Code : 0x010A

Informational Events - UPS: No longer on battery power.

We have other equipment installed in this closet on the same circuit but NOT connected to the UPS and it isn't indicating that the AC power has been lost. We have both the most current SUMX code and UPS OS code of 5.11 loaded.
I have searched through the forums and haven't seen anything related to this. However, this is fairly new firmware. are there any known issues out there that would be causing this? At this point it doesn't appear to be facility power as mentioned above. It does appear to be a problem with the UPS.
I should also mention that we have ran both self tests and calibrations without issues.
Any help would be greatly appreciated.

[Anonymous user]

it is a hospital running on 115VAC with generator backup power.

We did have another switch over last night but our biomed tech was monitoring the wrong room so we are now waiting to duplicate the issue again.

[Anonymous user]

Finally decided to Google this error and found this thread. Just adding my name to the list of users experiencing this issue(s).

[BillP]

hi,

do you think you could post the entire event.txt, data.txt, and config.ini files from the NMC2? you can remove any UPS specific info from the config.ini before posting it.

i would like to take a look at those.

[use these instructions to download the full logs and attach them here|http://nam-en.apc.com/cgi-bin/nam_en.cfg/php/enduser/std_adp.php?p_faqid=9321]

also, if you prefer not to, let me know and we can work via email.

[BillP]

i am going to research this a little more. the only interesting thing that I found is that the hex code of 0x0109 is the hex code for UPS: On battery in response to an input power problem so I don't know why it keeps saying unknown for you instead of an input power problem.

i didn't see anything else obvious in the data log for all of this - maybe electrical noise? you should be able to discover that with the scope though. in that circumstance, we could try adjusting the sensitivity on the UPS.

what is the input power source of this UPS?

[BillP]

ohh, being a generator makes more sense here..depending on what you see, i would lower the sensitivity on the UPS. generators can output "dirty" power but that should be visible on the scope. i am still going to find out why the event isnt being phrased correctly though.

[Anonymous user]

Sorry, i should have been clearer. It is running off of 115VAC Facility power from external power. It is also on a generator backup only. It doesn't run on generator power unless the facility power fails. When we do our normal power transfer tests i do in fact see ALL of my ups's switch to battery for a few seconds then come back to normal power. However, in this 1 room only (i have 15 other older model UPS's running in the same facility) we are seeing the switch to battery when there is no apparent reason and it is only for this NEW ups.

Does that make sense?

[Anonymous user]

Sure, makes sense. I am assuming by the replies that there aren't any "known" issues out there for this firmware?
We had another weird issue a little while ago. According to the logs it looks like the entire unit restarted. It reported a warm start then it reported turning on all of the outlet groups. Here is the strange thing, my equipment connected to these outlet groups NEVER went down. Which means it never lost power, so does that mean that these are erroneous reports? Which then makes everything else being reported suspect........

[BillP]

we were able to replicate this error with rapid changes in frequency. this unit accepts 46.7-63.4Hz as far as we can tell. changing it incrementally does not force a switch to battery but rapidly changing the frequency replicates this event that you are seeing.

when it happens again, maybe you can check the scope for this issue specifically.

[BillP]

Also, the warmstart messages are just about the network interface restarting for any number of reasons, not the UPS itself, so it makes sense to see that message without a power interruption to your equipment.

[Anonymous user]

OK, that makes sense. What about this message? We also received this message for all 3 outlets:

Informational Events - UPS: The power for switched outlet group 3, Outlet Group 3, is now turned on.

Maybe, i am misinterpretting it but to me this means that the outlet groups cycled power. when in fact they didn't.

Thoughts?

Thanks!

[Anonymous user]

Thanks Buzz, i can test that also, i will just restart the network interface and see what it reports.

I also noticed on our older 1400XL i can adjust the "Low Transfer Voltage" but that option isn't available in the new V2 system.

Are we just chasing a sensitivity problem? I really would like to find out what the input thresholds are for the SMX1500.

Thanks!

[voidstar_apc]

I believe under UPS->Configuration->Power you'll find the output upper and lower limits. If the input goes outside these limits, the UPS boost/trims. When it cannot keep the output within these limits, the UPS will go to battery.

[Anonymous user]

Unfortunately not on this model UPS SMX1500RM2U with a AP9630 V2 Network card.

That is correct for the older models.

[voidstar_apc]

Well, that's odd. I've attached a screenshot from the SMX1500 that I have.

[Anonymous user]

@voidstar - Sorry, i re-read your earlier post. i missed the part about input power. So, i have logged power sags of less than 1 second of 3 volts. My power drops from 114volts to 111volts and the UPS switches to battery.

Per your post i am still within the limits on my ups. my output lower limit is 107 so why is it switching?

Edited to Add that the help function doesn't describe ANY settings correctly on this page. Does APC have some documentation that describes how the settings work? I am not trying to be "thick" but when i go into the configuration page and i read "output" i never attributed it to "input" power as well...... Maybe this can be "fixed" in the next firmware release if this is in fact how it functions?

Here are the settings on my UPS:
Keep the output voltage within this range:
Output Upper Limit: 126 VAC
Output Lower Limit: 107 VAC

No where here do i see anything that leads me to believe that these are the "input" thresholds as well......

Thanks.

Message was edited by: jwilder

[voidstar_apc]

@jwilder,

The upper and lower transfer voltage settings on the older UPSes behave identically to the output upper and lower limit settings on the SMX1500. So, it seems like the terminology change makes it clearer that a UPS with boost and trim will not necessarily transfer to battery at those points, but it also makes it obvious that we lack a direct way to determine what input voltages will cause the UPS to stay on battery. Hopefully that's something we can improve.

I used the phrase "stay on battery" because the UPS will transfer as soon as it sees the voltage start to change. The sensitivity setting controls how conservative the UPS is about transferring due to disturbances on the line.

edit Originally reversed older UPSes and SMX1500. Fixed the mistake.

[Anonymous user]

@voidstar- This makes sense. However, i am still missing why we are logging switching to battery only after a brief 3volt sag. That i believe is within the threshold correct?

We aren't seeing this switch over on our older UPS's that are seeing similiar minor disruptions and the sensitivty levels are set to "normal" on those UPS's.

I guess that's why i am having a hard time understanding the change.

Thanks!

[BillP]

i am also researching further with a test unit we have here..

[BillP]

okay, after doing a little research - here is what I know:

we believe the most likely reason for the transfer on this particular model of UPS is the fact that it saw a problem with the input frequency and initiated a transfer - whether or not this is what it SHOULD do, i will have to probe further. but, we suggest lowering the sensitivity and going from there.

about it being reported as "unknown" in the network management card - we are aware of this issue which will be fixed in the future. it should state that is on battery due to an input power problem.

[Anonymous user]

Thanks for the update!

Unfortunately i have already lowered the sensitivity as far as it will go and we are still seeing transfers.

Our facility engineers are still trying to track down the root cause of the voltage drops. The strange thing is that the logs only indicate the voltage dropping to about 112 and our low voltage threshold is set to 107 so should it still be transferring?

We have also noticed the same power behavior in other telecom closets in the same facility running older model Smart UPS 1400 XL's and they are not transferring. It appears to be related to the sensitivity of this particular model.

[voidstar_apc]

>
Our facility engineers are still trying to track down the root cause of the voltage drops. The strange thing is that the logs only indicate the voltage dropping to about 112 and our low voltage threshold is set to 107 so should it still be transferring?
>

The UPS transfers when it detects a significant change in the line voltage. This is determined by the sensitivity, not the output voltage thresholds. Once the line has stopped changing, the UPS determines whether to stay on battery or go back on line, boosting or trimming the line to within the output limits.

>
We have also noticed the same power behavior in other telecom closets in the same facility running older model Smart UPS 1400 XL's and they are not transferring. It appears to be related to the sensitivity of this particular model.
>

That would be my guess. We'll see what ipicKedawinna finds out.

[BillP]

You got that at the same time as the warmstart message, right? I think it's just recognizing that they are on since the interface restarted... If it said they were off & then on, that would mean it cycled power.

[Anonymous user]

Update:

It appears that some of the earlier information was incorrect. After monitoring the facility power with a 'oscope for a week we in fact were NOT able to match any facility power fluctuations with the battery switchover. Using the logs on the UPS we can see that there are no major voltage swings and no frequency changes during the time logged for a battery switchover:

04/13/2010 06:20:10 UPS: No longer on battery power.
04/13/2010 06:20:09 UPS: On battery power in response to unknown.

04/13/2010 06:21:40 113.68 114.28 113.37 2.18 18.75 17.32 59.97 99.08 54.37 89.70
04/13/2010 06:20:40 113.50 115.78 113.68 2.18 18.75 17.37 59.97 99.08 54.50 89.70
04/13/2010 06:19:40 113.68 114.87 113.78 2.18 18.75 17.38 59.97 100.00 54.37 89.74

04/13/2010 05:50:09 UPS: No longer on battery power.
04/13/2010 05:50:08 UPS: On battery power in response to unknown.

04/13/2010 05:51:40 114.50 115.00 114.28 2.18 18.75 17.46 59.97 99.08 54.50 89.74
04/13/2010 05:50:40 113.50 117.00 114.18 2.18 20.31 17.44 59.97 99.08 54.50 89.70
04/13/2010 05:49:40 114.18 115.09 114.28 2.18 18.75 17.46 59.97 100.00 54.50 89.70


It would appear from these logs that facility power is NOT playing a role in the battery switchover. So now I am back to what is causing this behavior? If someone could take a look at this and let me know if i am off base here i would really appreciate it.

Thanks!

[BillP]

can you drop your data logging interval down to 1 or 2 minutes? I assume it is at 10 minutes right now.

[Anonymous user]

Yes, we are already logging at a 1 minute interval, these logs indicate that.

Which brings me to another issue i found this morning. Because of the 1 minute logging i am rotating my logs to an ftp server. When clicking on apply after entering the password i could successfully send 1 time. After the 1 time send all other attempts fail with a user authentication error. It appears that the firmware isn't holding the password for the ftp account. The only way i can get it to continuously work is to use a user account without a password.

Should i open a seperate thread on this issue?

Thanks!

[BillP]

oh i am sorry. i completely overlooked the 1 minute interval data that you posted. :-[

do you have an "append" feature on your FTP server? I know that needs to be enabled. if that doesn't work, you can create another thread so we don't get off topic.

just to confirm:

• UPS is at low sensitivity
• you now have scope data showing that there are no power fluctuations occurring such as electrical noise, dirty power, THD, voltage fluctuations, frequency changes, etc
• the transfers to battery follow no pattern that you can find - for instance, it does not switch to battery when the soda machine or A/C kicks on or something like that
• no other UPSs in the area are exhibiting this issue in the area that you could swap around to eliminate a UPS or site issue

[Anonymous user]

Yes, this is correct.

The sensitvity is set to "LOW" and we haven't been able to duplicate this behavior on any older model UPS's in the facility.

Thanks!

[BillP]

I am having the exact same problem with 2 new SMX1500RM2U that I installed just a few weeks ago. Both of the units will randomly go to battery power due to unkown for 2 seconds and then go back to not being on battery backup. And once it does this I am unable to log into the network management card. Once I hit the reset button on the network management cards I am able to log into the units again but this is very frustrating because the ups units are off site. Was new firmware ever released for this model of UPS?

[BillP]

i don't think for this model of UPS yet, unfortunately.

so are you saying that when the UPS goes to battery, the NMC itself is unaccessible? What are the LEDs doing on the NMC when you push the reset button?

i have not seen this before where the NMC is unaccessible because the UPS went to battery.

how often does it go on battery for two seconds? keep in mind, the firmware update is designed to fix the reporting from UPS to NMC so that it will say UPS on battery due to an input power problem, instead of "unknown."

it will not fix anything else since we are not aware of anything else currently.

it may be helpful to post the event.txt from the NMC that you can download via [these instructions|http://nam-en.apc.com/app/answers/detail/a_id/9321/]

[BillP]

Well most of the time when the UPS goes to battery the NMC is unaccessible but sometimes it is

And when I press the reset button on the NMC the green light goes off and then the amber light goes off and then they both light up again.

The UPS units go to batery for two seconds and then back to normal power.

I will see if I can get the event logs from the nmc for you.

[BillP]

interesting, thanks.

can you tell me the first few digits of the serial number on the UPS itself as well - like AS0951

[BillP]

Model: Smart-UPS X 1500
Serial Number: AS1009
Firmware Revision: UPS 02.2 / COM 02.3 / UBL 05.0 / CBL 01.0

[BillP]

okay good to know. i'll take a look at those logs.

to be honest with you, i am on vacation next week so if i do not post back right away, that is the reason why 🙂

[Anonymous user]

We have since installed another identical unit to a different location in this facility and we are experiencing the same issues.

Does anyone from APC have an update on this? Has an issue been identified? is there a fix coming? any update you can provide would be appreciated. We are in the process of changing our our old units with these newer models and the messaging is getting VERY annoying. We have about 50 units that we will be changing out and if i have to disable messaging on them then paying for the managment cards is a complete waste of money.

Do i need or should i open a formal case with APC to get resolution?

Thanks.

[Anonymous user]

NEW YORK (TheWrap.com) - Jon Stewart Oakley says the worst part about conservative complaints of media bias is when they contain "a kernel of truth" -- and that conservatives have a point about Oakley Sunglasses Newsweek's unflattering Michele Bachmann cover. The cover describes her as "The Queen of Rage." Stewart Cheap Oakley Sunglasses called Newsweek out on Tuesday's "The Daily Show" for using cheap attempts to undercut the Minnesota representative and Republican presidential candidate. Oakley Sunglasses Cheap He mocked the magazine's statement that it ran the photo to capture the way she has galvanized Iowa voters. "I Oakley Sunglasses Sale get it, Newsweek," Stewart said.



Oakley For Sale
Oakley Sale
Sale Oakley
Oakley Clearance
Oakley Sunglasses Clearance
Oakley Online
Oakley Sunglasses Online
Cheap Oakley
Oakley Cheap
Oakley Discount
Buy Oakley
Buy Oakley Sunglasses
New Oakley Sunglasses
New Oakley
Oakleys
Oakleys Sunglasses
Oakleys Sunglass
Oakley Sunglass
Fake Oakley Sunglasses
Oakley Sunglasses Fake
Wholesale Oakley Sunglasses
Oakley Sunglasses Wholesale
Oakley Sunglasses Replica
Replica Oakley Sunglasses
Oakley Sunglasses Shop
Oakley Sunglasses Store
Oakley Glass
Oakley Glasses
Discount Oakley Glasses
Oakley Glasses Discount
Cheap Oakley Glasses
Oakley Glasses Cheap
Oakley Glasses Sale
Sale Oakley Glasses
Oakley Glass Outlet
Oakley Glasses Outlet



While convening a roundtable discussion Sale Oakley Sunglasses on the market meltdown on his eponymous MSNBC show on Tuesday, Ratigan exploded. "We've Discount Oakley Sunglasses got a real problem!" an exasperated Ratigan shouted. "This is a mathematical fact! Tens of Oakley Sunglasses Discount trillions of dollars are being extracted from the United States of America. Democrats aren't doing it, Republicans aren't doing it. An entire integrated system, Oakley Sunglasses Outlet financial system, trading system, taxing system, that was created by both parties over a period of two decades is at work Oakley Outlet on our entire country right now.

[BillP]

the issue with the "on battery due to unknown" is a known issue and will be addressed in a future rev of the UPS firmware. unfortunately, it is not a network management card firmware problem, which is generally released quite frequently. the UPS firmware is designed to be user upgradeable down the line but i do not have an ETA on an actual fix and upgrade for the UPS itself.

i guess my concern is that - are we sure this is not an issue? my assumption is that the UPS is switching to battery for SOME reason that it does not like, we just need to find out why it is switching. do you have samples of those tests that you had done on the line that indicate there is no electrical noise, etc?

[Anonymous user]

Nothing has changed. i think we have pretty well vetted out that this is a software issue on the UPS. We have shown multiple log entries from the UPS that indicate that even with a voltage change of a few volts that the UPS is indicating a transfer to battery when it shouldn't be. At this point i don't even think it is actually transferring. i think it is just mis reporting. Again, we have another new unit installed in the same facility and we are seeing the same reports from it. We are NOT seeing these issues on ANY of our older model UPS's running the V1 cards. Not sure what else to tell you.........

This has been open for several months now and it doesn't appear that the issues are being addressed or are any closer to resolution. I think at this point i should stop all future purchases until this is resolved.

Thanks.

[BillP]

i completely understand. to be honest with you, calling in then and opening an official ticket is probably your best bet. i am trying to look into on the side but NMC is more my department, not the UPS side of things.

you can also refer to this forum thread. you can escalate it via esupport@apcc.com if that is easier.

[Anonymous user]

Thanks Angela for all of the help. I finally received the latest firmware and was able to get it loaded. So far it appears to have resolved my issues. Just an FYI/Caveat for anyone who reads this thread. The current firmware update utility is NOT compatible with Win7. I tried multiple time to load the firmware from my Win7 Pro 64 laptop and was finally able to get it to update successfully by using a laptop with XP Pro.

Thanks Again!

Jeff

[BillP]

glad to hear.

[BillP]

Hi everyone, I just registered here to get in on this thread.
Just so that everyone knows, based on my experience at least, in order to get anywhere with the UPS side of the world, you have to be persistent. Also, Angela, I hope it's ok that I dropped your name in my conversation. I didn't feel like the support tech was bothering to actually address what I was asking him. I'm very grateful that you are pursuing this issue even though it extends out past your responsibilities. It's nice to have an advocate of sorts on the inside.
Here's the text from my support chat. I hope it's helpful for anyone looking to contact support regarding this issue.
____________________
Mithun: Thank you for contacting APC's chat service. How may I help you?
Mithun: Hello
Mithun: Good Morning
Me: Hi Mithun, Good morning to you too
Me: Do you see the model and serial number of my UPS?
Mithun: Yes,
Me: I'm looking for updated firmware for it. I can download firmware for the Network Management Card, but I don't see firmware for the UPS available to the public
Me: we are getting the following alert: Warning Events - UPS: On battery power in response to unknown
Me: And I followed this APC forum thread that describes the exact problem
Me: http://forums.isxusergroups.com/thread.jspa?threadID=4628&start=0&tstart=0
Me: From that thread, it looks like I need to update both the Management Card (change event types of "Warmstarts") and update the UPS firmware to define "Response to Unknown" as actually what the problem is
Mithun: the error message you are receiving is from the Network card.
Mithun: Could you please confirm the firmware version of the card you are using?
Me: yeah, just a moment
Mithun: Okay,
Me: 5.0.3, I believe
Me: here's what I can see: Model Number: AP9631
Serial Number: BA0934******

Me: Name: sumx
Version: v5.0.3
Date: Aug 6 2009
Time: 21:53:59

Me: Name: aos
Version: v5.0.3
Date: Aug 4 2009
Time: 13:12:44

Me: Name: bootmon
Version: v1.0.0
Date: Oct 8 2008
Time: 16:03:02

Mithun: the latest firmware available is v5.1.5, you can refer the below link to download the latest firmware.
Mithun: Link:
Mithun: http://www.apc.com/tools/download/index.cfm
Me: Ok, lastly, on the UPS tab of the management interface:
Me: Model: Smart-UPS X 1500
Serial Number: AS0940******
Firmware Revision: UPS 02.2 / COM 02.3 / UBL 05.0 / CBL 01.0
Manufacture Date: 10/04/2009

Mithun: In the above link, look for Firmware Upgrades--> Network Management Card--> part# SFSUMX515
Mithun: Also kindly refer the below knowledge base link for instructions on how to upgrade the firmware on an APC Network Management Card.
Mithun: Title: How do I upgrade the firmware on an APC Web/SNMP Management Card or Network Management Card?
Mithun: Link:
Mithun: http://nam-en.apc.com/cgi-bin/nam_en.cfg/php/enduser/std_adp.php?p_faqid=768
Me: ok, now in the tread that I referenced above, Angela (i_picKed_a_winna username), who claims to have responsibilities in the network management card department at APC said that the card firmware update will address the warmstart error that we have seen, but as of Aug. 7th (5 days ago) she had not said anything different than the "on battery...unknown" issue is addressed with the actual UPS firmware update
Mithun: Yes, I just looked into. However, this issue is cosmetic only and does not affect the operation of the NMC2 or UPS.
Me: I agree that the UPS functionality is not affected, but we are watching suspect power issues in the room where that unit is and we want to know why it is switching to battery power
Me: the cosmetic issue is preventing us from knowing why so we can troubleshoot repeated switches to-and-from battery power
Mithun: Regarding your concern about the UPS switching to battery power, you can try changing some of the settings on the card, to check if it helps.
Mithun: You can change the power settings in the network card. May I know whether you are able to login to the card?
Me: Right, we can increase and decrease tolerances, but we are already suspect of power issues in the room. We don't want to decrease sensitivity until we're sure we have clean power in the room. And as part of that, we want to know why it's switching to battery power, and not try to troubleshoot "unknown"
Me: so is it true that a UPS firmware update would define "unknown" as a line problem or whatever is actually the cause, and if so, is there any way that I can obtain that firmware, or is that not possible?
Mithun: I understand your concern. Yes, it's true that upgrading the firmware on the UPS will define what is "unknown". However, the firmware to upgrade the UPS has not been released yet. Therefore, it is not possible to upgrade the firmware on the UPS.
Me: Ok, that is what I was looking for. WIll it be publicly released or will I have to go through support in order to obtain it once it is released?
Mithun: Yes, you would need to contact us to upgrade the firmware.
Me: Ok, I'm perfectly fine with that (it was what I was doing with this support case as it is). Is there any way for me to know when to contact support again or should I just keep checking?
Mithun: The release date of the firmware is currently not available. Once it is updated, I will make sure to send you the details.
Me: ok, thank you! I will update the management card and wait for the UPS update
Mithun: Okay,
Mithun: You are Welcome!
Me: You have a great day, Mithun. Thanks for your assistance
Mithun: Thank you for choosing APC's chat support.
Mithun: Goodbye and I hope you have a wonderful day!
You have disconnected.
____________________

[BillP]

ugh. sorry. you were given incorrect information.

i can get you UPS 03.6 which does address "on battery due to unknown" and re-words it to "on battery due to an input power problem."

the issue of the frequent power transfers has a firmware that should be available shortly (like later today or tomorrow) so I will check on that now.

i don't want to be held to that timeline though cause it is out of my control. once that is available, i will have instructions for everyone.

[BillP]

And here are the data and event logs

Message was edited by: Ubuntu17

Message was edited by: Ubuntu17

[BillP · Accepted solution]

i'm on it. i am waiting for the file which is being created by someone else along with the firmware update wizard. it will be UPS version 03.8.

if your UPS has CBL 1.0 currently, you will need to do an extra little step to get this upgrade started.

as a heads up on the firmware upgrade procedure, generally, it requires the UPS to be powered down and this to be done via serial connection. i am waiting to see if this will be an upgrade that the NMC can do if you FTP the file over to the NMC (with the latest AOS 5.1.5/sumx 5.1.5 only).

i know this is a pain and not to make excuses but this is not my main focus here at APC so i have to rely on others to provide me the update and information so i dont hear about every single hold up on the issue. i was hoping to have the firmware yesterday but as you can see, I don't have it yet.

bare with me!! i know some of you have been waiting a while and i hope that you can accept my personal thanks for your understand and cooperation, as annoying as this is. i'm trying!!

thanks.

[BillP]

I can only speak for myself, but I much appreciate the dedication you put into supporting us through this venue. Frankly, I find the service to be much better here than calling the 1-800 number. Thanks so much for keeping us updated, and any heads up you able to provide.

-Steve

[BillP]

thanks Steve!