Freezing advanced OPC analog points

[EC_andgus]

Hi,

I’m facing a problem with freezing advanced OPC analog points in Geo SCADA.

The server is set to OPC DA 3.0 and asynchronous read/write. Both analog and digital points are connected to the same OPC group. The group is set to asynchronous, 1s update rate and confidence polling, 1s interval.

I have this issue on different server versions but only with the advanced OPC driver. This was never a problem with the simple driver.

Usually this occurs after a couple of weeks from server restart when the database memory usage is high. The digital points still works fine and after server restart the analog points works fine again.

I would appreciate some inputs from you if you have the same experience and what I can do to make my setup more stable.

Should I split the signals to more OPC groups?

Should I activate confidence polling on each point?

/andgus

[BevanWeiss]

If you believe there is a 'bug' with the Advanced OPC driver that is causing it to 'freeze' points every so often, then you really need to be collecting logs (both Server Logs, Advanced OPC Driver Logs, and logs for the remote OPC server where possible), and sending them through to Tech Support to resolve the core problem (if there is one in the Geo SCADA Expert product.. inc Advanced OPC Driver).

I can't say that I've seen this same issue, but then my usage of Advanced OPC points is very limited.

OPC-DA itself is often quite troublesome, so I generally try to avoid it if at all possible.

I would say OPC-UA would be the better way forward... but alas, Geo SCADA Expert still doesn't have support for OPC-UA natively.  Which does remind me... I'll confirm that there is a listed enhancement here for this (OPC-UA)

[Anonymous user]

If you've got it set to asynchronous, then you're relying on the device you're connecting to sending updates asynchronously.

Having 1s update rate and 1s confidence poll seems pointless and inefficient.

If memory usage is high, maybe the server needs more memory?

But yeah as Bevan says, any deeper than that should probably be raised with support with logs.

OPC UA is coming - I think I'm allowed to say that!

[EC_andgus]

Not really sure how the confidence poll works… Will look at it.

The machine has enough memory. It’s the memory usage of Geo SCADA database server and OPC driver that grows fast, it grows about 100/50mb per 24h and when it use 1500-2000mb this happens with the analog points.

The screen dump is from a 352 database point server running for 5-6 weeks.

I have raised this issue with the support.

/andgus

[BevanWeiss]

That seems like an awful lot of memory (both DB Server, and especially OPC Driver) for only 352 points.

Are you doing anything 'tricky' with Logic or similar that would be altering the database frequently?

What settings do you have for your Significant Change entries? and how many historic records are each point generating every minute?

I'm also intrigued by the Advanced OPC Driver being shown as 32 bit.  Perhaps I've never looked closely at this driver in other systems.  But my expectation is that this would be a 64 bit executable.  I have just checked, and 32bits it is.

This does have the potential to cause problems if there are lots of OPC items which result in the memory usage of the OPC driver exceeding the 32bit memory space.  It seems that this should be quite unlikely however..

@Anonymous user any chance you could raise with the Devs as to why the Advanced OPC driver is still listed as 32bits on a 64bit OS?  I can only imagine that it's for compatibility reasons with some InProcess 32bit references (maybe the linked OPC-DA protocol stack..).  I would have expected there would be a 64-bit option available for this however, and for larger point counts (or high memory usage scenarios) 64-bits would be a 'safer' option.

@EC_andgus if you're polling so rapidly, do you need Asynchronous?  Obviously it's a better situation, but you may be running into some issues with it, and having just fixed Synchronous 'scanning' might be a suitable workaround in the meantime.

[Anonymous user]

There's a few drivers that are still 32 bit, eg Crystal, Simple and Adv OPC.

Crystal is due some development to support CR2020, which now native x64. ViewX is still 32 bit, so the integration there will be interesting.

For OPC I imagine you're right and it's around opcenum being native 32 bit. We'd rather spend the dev time on OPC UA than porting the existing drivers to x64 which from a little google seems to raise a bunch of extra problems.

[BevanWeiss]

Crystal Reports Driver is definitely 64bit on my machine.

There is a limitation on the Crystal Reports Designer, only supporting Crystal Reports 2016 since 2020 went to 64-bit only.

This relates to ViewX only being a 32-bit app I believe. 

Apparently this does OPC-DA for both 32-bit and 64-bit

https://github.com/shushudu/OPCDAClientSync

and MIT license 😉

Although InProc is going to be an issue in either case... Since an InProc 32bit OPC DLL can only be loaded into a 32bit parent process, and equivalent for 64bit.  Perhaps there needs to be two Advanced OPC Drivers... one for 32-bit, and one for 64-bit.. with 64-bit handling all Local and Remote servers, and 64-bit InProc servers, and the 32-bit driver only handling the 32-bit InProc servers.

It does seem a bit more painful than OPC UA however... 

The Licence Server... meh, it doesn't really matter if it's 32bit or 64bit, but I do wish the spelling would pick whether it was going to go with English or US through out 😉

The Server License Details dialog is a fun time.. as is the help.

[AndrewScott]

Confidence polling is used to prevent data from becoming stale by forcing points to be updated periodically, see https://tprojects.schneider-electric.com/GeoSCADAHelp/Geo%20SCADA%202020/Default.htm#AdvancedOPCDriv... 

It seems unlikely that users would loose confidence in the point data after only one second if it hasn't been updated. Confidence polling is typically set in hours or days not seconds. 

[tfranklin]

Are your analog points configured for any kind of significant change?  In advanced OPC with async scan, analog values only update if you force it with confidence polling (don't do that) or have significant change applied.  Jessie might be able to give more insight but I suspect that because of this being a 32 bit driver as mentioned above, they're enforcing strict configuration on the points to relieve unnecessary stress. 

When we set up OPC we almost always use async polling, no confidence polling, and apply significant changes to all analog points. 

I'd suggest using confidence polling similar to simple OPC background logging.  Only use it if you need to force an update from the OPC server device/cache, else it should never really be used. 

[EC_andgus]

I didn’t have significant change with the asynchronous setup because of this note in the manual:

“The Significant Change Properties are only applicable when you are using synchronous polling. The Deadband (Asynchronous) feature provides similar functionality for asynchronous mode”.

I have change to synchronous polling with significant change on every analog point on one site with 4000 OPC points for evaluation.

/andgus

[tfranklin]

Interesting.  I always leave the deadband on the scan group for async = 0 because it's only a % deadband.  We find most customers want absolute changes which would require point by point configuration. 

I can say with full confidence that async updates are affected by the significant change settings despite the documentation saying it's for synchronous only.  Test it out and see if that ends up resolving your issue, even if it's just a handful of points.  If you already have the points set with sig change, flip it back to async updates and you should be fine. 

For reference -- we have installs of systems ranging from 10k-300k OPC points and the mentioned configuration has yet to let us down.  Memory is typically not an issue either. 

-----------------------------------------

I just ran some quick tests and had DA3.0 with async polling against my OPC server (AUTOSOL/ACM).  I created 3 analog points: 1 no sig change/deadband, 1 with a sig change of 5, and 1 with a deadband of 5%. 

I forced updates to a modbus simulator ranging from 0-100 and the only value that changed to accurately reflect the updates  was the one with a sig change of 5.  I believe the documentation is incorrect here. 

There also appears to be something funky with the deadband setting.  I have my settings at 0-100 and I forced a value from 5 to 75 with a deadband of 5%. No updates. 

My suggestion -- unless I'm crazy and somebody can talk me off of it -- if you use async polling then ignore the documentation and set a sig change and act like confidence polling isn't a thing.  It works.

[AndrewScott]

FYI, the OPC DA 3.0 deadband percentage is applied by the OPC server (using the IOPCItemDeadbandMgt::SetItemDeadband() method), so if the deadband isn't being applied correctly then there may be an issue with the OPC server.

The OPC DA specification (section 4.4.9, page 106) describes this feature as follows:

"This interface allows the PercentDeadband to be set for individual items within a group. Once the item PercentDeadband is set, it overrides the PercentDeadband set for the entire group. This provides a mechanism to set the PercentDeadband on a “noisy” item, which may reside in a group that doesn’t have the group PercentDeadband set. It also allows individual items to be fine tuned with respect to notifications based on an expected range of change."

[tfranklin]

Noted and verified on my end! Looks like we don't have that implemented in our driver.