Sluggish Value Item Creation/Updates

[Lydia]

One of my company's clients have around 3,500 points (or value items) which communicates between our company's software (CS) and Struxureware (SX). I've built a Smart Connector processor to communicate between the two. When the number of points are relatively small, my processor performs adequately. As the number of points increase, however, so does the time it takes to initialize/update container items and value items. Of course, this makes sense: As the number of points increase, so does the time it takes to perform all necessary operations. The problem, however, is that updating all value items can take minutes at a time. This time delay is unacceptable for our use, unfortunately. 

I have performed numerous tests on my processor, and the results are as follows:

- Obtaining container information from CS - 2 seconds

- Obtaining item information from CS - 2 seconds

- Processor item creation/calculations - 0-2 second(s)

- Creating containers in Struxureware - 12-15 seconds

- Creating/Updating items in Struxureware - 1.5-3.5 minutes

The majority of the operating time is during item creation and updates. 

Simply commenting out the following line of code during the execution of the first run of the processor lets the processor fully run in <20 seconds.

DataAdapter.AddValueItem(altId, name ?? altId, description, type, writeable, forceable, defaultState, unit, parent);

I've been trying to think of different ways to counteract this problem such as 1) making the method for item creation/updates async, place the method as an action in a list of tasks, and start those tasks as separate threads (Didn't work, received errors), remove "async" from the item creation/update method and run those tasks as synchronous (which did not improve performance), and creating value items and appending those items directly into the EwsServer.ValueItems IEnumerable List (which did nothing).

I don't know what I can do to make item (and less importantly, container) creation/updating faster. 

[JeffBowman]

Hi @Lydia ,

So you are creating 3,500 objects in a Smart Connector EWS server? During the creation of these objects and updating their values it is taking longer than what is expected?

A couple questions.

- Isn't the creation of the objects only a one time thing on the initial run? So this 'delay' should only be on initial creation. I am curious as to why this is not ok. Before I go and tell you how to insert all the new objects in a single transaction (which is risky because there are many checks that are not done when doing it this way, I want to understand the reason for this problem because there might be another option)

- As far as value updates specifically, there are ways to speed this up. Mainly you can turn off AutoCommit and commit all the value changes at once using something like the code below. (NOTE: This only works for property changes, and NOT object creation).

DataAdapter.AutoCommit = false

foreach (var item in itemsToUpdate)

{

DataAdapter.ModifyValueItemValue(item, value, state);

}

DataAdapater.CommitChanges();

DataAdapter.AutoCommit = true;

Best Regards,

-Jeff

[Lydia]

I apologize for the confusion. Since it is only performed once, item creation is not a very large problem. The lag in updating the items is.

My processor already has AutoCommit turned off. I changed my processor back to its original state where it performs all the updates and commits the changes once at the end of the processor run.

These are the times I have right now:

- First run: 5.6 minutes

- First run + Second run: 7.7 minutes

As you can see, the second run - which updates all the containers and items - takes 2.1 minutes. I can scale it back to 1.8-1.9 minutes by not checking any changes to containers. But needing 2 minutes to update all points (approx. 30 items/sec) still takes awhile simply due to the number of points.

Is there anything else I can do to make the processor run faster?

[JeffBowman]

Hi Lydia,

I think it would be best if i could perhaps see the code. Feel free to email me at jeff.bowman@se.com, I am available the rest of the afternoon today, or we can schedule another time for me to take a look if you are interested.

Best Regards,

-jeff