Panel Server - Known bugs by firmware

[Sam_Vosch]

Hello All,

To prevent others from falling into the same issues as we have with this range of devices, I have compiled a short list of bugs in the PAS firmware that we have encountered. These bugs are centered around file drop, accuracy of file timestamp logging and ETL. 

Firmware 1.7.0

This firmware changes the measurement naming for the IO devices, namely Smartlinks. This will break ETL integrations to PME and PSO. We have not confirmed whether this affects custom devices from the portal. 
For Example:

Old measurement name:   IoCountMeasurement (L)

New measurement name: (L)

Solution: Modify ETL mapping for the new measurements.

Firmware 1.6.0

1. NTP functionality is working but not reporting in the web page correctly. Web page Reports 'OK' but the 'Last Date and Time Synchronization' does not update making it appear like it hasn't synced in a long time. Looking into the logs we can see it is syncing. The .CSV file drops are correctly timestamped but are in UTC+0 timezone.
2. The .CSV format issue. The PAS .CSV file includes "" around every field. This causes all sorts of issues to .CSV extractions tools such as ETL. This was introduced in 1.6.0.

Firmware 1.5.1

1. NTP functionality not working. Reports 'OK' but never syncs time. Even when a completely random manual IP is used it reports 'OK' 🙂
2. When a SmartLink is added, SFTP fails after 1-3 weeks. Appears to be centered around the PAS losing SmartLink settings (according to the PAS logs). Reboot will make file publication work again BUT without some SmartLink device names in .CSV file. Data extraction tools like ETL cannot process the files without a devices name.
3. When the PAS network is set to 'Separated', the published file do not have a MAC therefore no valid start name for ETL to reference (you cant use ETL). Solution - update to newer firmware.

Firmware 1.5.0

1. Same bugs as 1.5.1

NOTES:

A temporary work around for the 1.6.0 .CSV " format issue can be achieved using a PowerShell script and Window Task scheduler. It's not ideal and may be prone to fail when windows update.

Other minor issues:

• Still no native support for Schneiders PM5350 meter after many firmware releases 😞
• The Time zone settings limited to 0+ UTC. Difficult to work with when you don't live in 0+ UTC
• SFTP File publication only supports 1minute, 2 minutes, 10minutes and 1hour. There is no 15minute interval which is a standard logging interval.
• When manually exporting data from the PAS800, the .CSV file format is incredibly hard to work with compared to the ComX510. Most customers cannot use the .CSV exports as the Panel Server in the way the marketing material suggests, but require either Energy Hub or PME to work with it.

[UdhayakumarAK]

Hello @Sam_Vosch 

Thanks for highlighting these points. For NTP time sync issue, have you raised any customer support case? If not would recommend you to raise a support ticket with logs & screenshots.

For csv file format issue, as you mentioned temporary solution is to use the PowerShell script & this will be addressed at system level in ETL tool hotfix releasing this month end.

https://www.se.com/in/en/faqs/FAQ000259575/

Thanks & Regards

Udhay

[Sam_Vosch]

Hello UdhayakumarAK, thank you for your reply.

I raised the issues with Power Solutions Global Expert Technical Support weeks ago (logs included), and you replied to the issues. As you know from my latest reply, some of these issues are still ongoing as we find out more about the issues.

The main purpose of this post is to inform the wider community of the issues with the PAS devices. I have not seen a bulletin or post informing the community of the community of serious firmware issues that will compromise data into their PME/PSO systems. Hopefully this community post will help other Schneider partners and prevent others finding the same issues the hard way.

[Sam_Vosch]

Further point on firmware 1.5.1: 

We found that Firmware 1.5.1, while it also has the same sync issues as 1.6, it does not add the UTC time offset to the .CSV file.

This can cause serious erroneous data entries into customer sites. Essentially, if ETL or similar applications are used, it will input the data 12 hours up to into the future (if you are in +12 GMT). 

Special care will need to be taken if the site uses consumption data for billing purposes.

[Guillaume_Evrard]

Hello NTP is working well using 1.6, why are you saying it is not ? There was an issue with the SE NTP server, that should have been fixed.

[Sam_Vosch]

Hello Guillaume,
I have been working with support and it looks like NTP functionality is working but not Reporting in the web page correctly. Web page Reports 'OK' but the 'Last Date and Time Synchronization' does not update making it appear like it hasn't synced in a long time. Looking into the logs we can see it is syncing. The .CSV file drops are correctly timestamped but are in UTC+0 timezone.
I will update the original post.

[Eric_HAVAS]

Hello Sam,
It s been reported in web page but only when time difference between PaS time and ntp time is signifiant.
Regarding CSV, the "" were added following customer complains regarding "special" characters used commonly in their country causing unexpecting behaviour in csv export of SFTP transfer.

[CamboHeath]

I am encountering numerous issues with the NTP in both PAS600 and PAS800 gateways on Firmware 1.6.0. that line up with what Sam was seeing above. Can you please advise whether these issues are on the radar for resolution?

On one site i have several PAS800s and PAS600s that all say "Connection successful to NTP server", however I can see they are out of sync by at least several seconds, and due to daylight savings they are now an hour out of sync even though they say that the connection is successful. However on another site I will get "Connection failed to NTP server" regardless of if the devices sync to a NTP or the Schneider electric cloud. 

Please advise what the story is with getting NTP functioning and reporting correctly on Panel Servers.

[Guillaume_Evrard]

Hello,

Issues with NTP have been reported with 1.7 also. We had an issue over the last weeks with the SE NTP server (one used by SE Cloud) not responding and sometimes using an alternate NTP server can solve the issue. Remember to whitelist the url.

Normally you should be able to see the synch message in the log periodically. Today PAS is using UTC only, so not sure to understand the point with daylight saving,

[CamboHeath]

Thanks for the reply Guillame. Just to confirm:

1. Should SE NTP currently be fully functional on 1.6?

2. Should Local Static NTP currently be fully functional on 1.6?

I have several PAS's on 1.6 all pointing to the same static NTP that all say "Connection successful to NTP server" but that clearly are not synced when a new minute rolls around on the NTP source and they all tick over at different times. None of my PAS's syncing to the SE NTP appear to be syncing either

Can you please confirm if the NTP should be fully functional on 1.6? Thanks

[UdhayakumarAK]

Hello @CamboHeath 

As mentioned by Guillame we had SE cloud NTP time sync issue in past weeks, and still there are some interim issues ongoing with cloud sync. And with respect to static NTP, we don't see any gap. Please raise a customer support ticket for further investigation with below details

1. Screen Capture of the time difference with respect to NTP server

2. Screen Capture of PAS Date & Time settings page 

2. PAS logs (both diagnostic & configuration logs)

But strongly suggest to upgrade the Panel server to the latest Firmware version.

Thanks & Regards

[Sam_Vosch]

I have updated original post to include issue with Firmware 1.7.

[Sam_Vosch]

I think we have solved the NTP issues. When using a local static NTP server on a local network (such as 192.168.1.x), on either a closed managed network or an open unmanaged network, the PAS MUST have a static DNS address?

To me this is so strange. We are using a fixed local IP address which would never need a DNS to translate the IP to an IP. A DNS is mostly used to translate an address like 'google.com' to an IP address. It should not be needed or related to the issue, but I have confirmed that setting the DNS to the router's IP fixed the issue on 3 different sites.

The PAS will not even attempt to send a NTP request on the network when the DNS is not set statically (as proven when we used Wireshark on the network). But as soon as the static DNS is set, we can see the PAS sending NTP requests in Wireshark.

[Guillaume_Evrard]

Hello,

From what I understand the NTP server set in PAS is a static IP address and not a url. Correct ?

I understand also that setting an IP address (router or gateway adress)  even not being a DNS is working ?

Fully agree with you it should work, do you know if a defect has been raised ?

Kind Regards

[Eric_HAVAS]

Hello Guillaume,
Yes before this post in se.com, Sam has already contacted the Customer Care Center, as it is the usual process for technical issues and the CCC took already into account his request which is currently being investigated by the panel server team.

[UdhayakumarAK]

Hello Sam @Sam_Vosch 

This is addressed in FW 001.009.000, suggest you to update the FW to resolve this issue.