External program using smartconnector stops

[mayumi_UWL]

I have a C# program that uses SmartConnector to log SBO info into MSSQL database.

It works for few days and suddenly stops logging.

Not sure what the issue is.

Other program seems to be working fine but just the logging stops completely.

What could be the cause of this kind of issue?

Server CPU?Memory?

Throttling Policy setting?

Throttling Policy

Attached a log file from smartconnector.

I don't know if it's the program issue or setting issue or server issue.

Would be nice if you could guide me to the right direction.

Thanks,

Mayumi 

[JeffBowman]

Hi Mayumi,

I am not fully sure I understand what you mean by 'it stops logging'. Do you mean that data stops being transfered, or Smart Connector stops logging stuff in it's log?

From looking at your smart connector log, it does seem to be spitting out the same error over and over again, but really there is nothing in the error that tells me exactly what is going on.

Can you change your logging level in the Smart Connector portal to 'Trace' (it is in Setup > Service Settings). Also if you could please turn on ' Ews Consume' logging in Setup > Logging Filters, this would be very helpful in determining what may be going on.

Also I don't think this is a throttling issue, but you can actually just turn off throttling all together, but simply deleting all the values in the throttling policy (e.g. change '20' to '').

Regards,

-Jeff

[mayumi_UWL]

Hi Jeff,

We have all of the below applications installed on 1 machine

• EBO
• Security Expert
• SmartConnector
• HVAC control program in C# (as Windows service)
• MSSQLDB

Server hardware spec of the machine

• Windows server 2012
• CPU: 2.2 GHz
• Memory:16 GB

Hard disk: 126GB (C drive)

"HVAC control program in C#" is our custom built program that controls EBO outputs through SmartConnector.
Whenever the output value changes On/Off by the program, it logs into MSSQLDB.

The problem we are having right now is, this custom logging to DB seems to stop working after a while.
Not the SmartConnector logging.

We do tend to get high CPU usage on our machine so maybe that needs to get fixed.

Wasn't exactly sure what should be the hardware requirement when you install all the applications in 1 machine.

But if you could find anything in the SmartConnector log that tells other issue, I really would like to know,

I have the log of SmartConnector after changing the setting from 'Error' to 'Trace' but the file size is too big to upload here. 
(Ews Consume was already turned on)

How can I send you the file? (It's about 30 MB)

Let me know if you need further clarification.

Regards,
Mayumi

[JeffBowman]

HI Mayumi,

If you are able to add the log file to box and send me the link,that works (you could try and email it to me too if you would like jeff.bowman@se.com)! I would like to inspect the log, and see if we can figure out if something is happening on the Smart Connector side to stop data being trasnferred to your SQL database.

Regards,

-Jeff

[JeffBowman]

Hi Mayumi,

I took a brief look at the logs, and I am not seeing what I thought I might, but I do have some questions about what your C# Program is doing.

1. How many different points are you pulling data from EBO?
2. How often are you pulling the data from these points?
3. What data are you using from each of these requests?
4. What REST gateway endpoints are you using to get this data? (I see likely you are using /Values/{id}/Value)

I know there is a known issue with EBO that if you send too many EWS requests in a short period of time (and EBO has a certain configuration), than EWS requests will stop working (although I don’t see any indication of this, this could still be the case).

Based on what I do see though, I think there are 2 things you should do:

1. Instead of using GET /Values/{id}/Value to get the value of objects you want. Use the /Subscriptions and /Notifications endpoints together to subscribe and read points. This is effectively batch reads, and is much much much more efficient for both your code, and EBO handling of the EWS requests (ill add this process below this email).
2. If your REST Endpoint is using the ‘http’ and not ‘https’ EWS endpoint for EBO, I would suggest changing this to https. This is because they made a change where ‘redirect web clients to https’ is on by default, and doing this effectively makes it so that http EWS requests need to consume a session for every request. I could get into the details if you want, but basically what this ends up meaning is that you can only send 1000 requests in a 5 minute period before your requests get a 400 response until the sessions expire. The alternative to changing to https would be to uncheck this box in Tools > Control Panel >Security Settings like in the below screenshot:
   

Here is the process for using subscriptions with the REST gateway in summary:

1. Create a Subscription.  A Subscription defines what data will be monitored by the underlying server.  Data is returned in the form of NotificationItems for a particular Notification.
2. Create a Notification.  A Notification is a session of shorts or a snapshot in time of all data changes.  This allows the server to manage paging of the NotificationItems for an arbitrary long number of data points.
3. Read the NotificationItems.  A NotificationItem is the actual data that has changed for a Notification of a Subscription.

The RESTful Gateway API (and data constructs) it uses was defined to support the full EWS specification, In the case of the SoapEwsRestProvider (at least when SBO is the server) you can ignore anything not related to "ValueItemChanged" subscriptions.

Here are the above steps in more detail:

1. Create a Subscription.  Issue a POST to Subscriptions/Create with the new subscription definition in the body.  Using Swagger, you will see the format of this body parameter:

{

  "DurationInMinutes": 0,

  "SubscriptionType": "ValueItemChanged",

  "Ids": [

    "string"

  ],

  "PriorityFrom": 0,

  "PriorityTo": 0,

  "Types": [

    "string"

  ]

}

Ignoring what you don't need for the SoapEwsRestProvider brings us to...

{

  "DurationInMinutes": 30,

  "SubscriptionType": "ValueItemChanged",

  "Ids": [

    "<< One or more unescaped EWS ValueItem IDs go here>>"

  ]

}

NOTE: Even though you can supply a DurationInMinutes value, SBO will always make subscriptions 30 minutes in length.

The POST result is the Subscription which you will need to create Notifications.

1. Create a Notification.  Issue a POST to Notifications/Create with new Notification definition in the body.  Using Swagger, you will see the format of this body parameter:

{

  "SubscriptionId": "<< ID of the subscription returned from step 1 >>",

  "ChangesOnly": true

}

ChangesOnly indicates that you only want what has changed since the last time you called created a Notification for a specific Subscription.  The first time you call Notifications/Create for a Subscription this is ignored and all data will be returned.  Subsequent calls can either return all data (ChangesOnly=false) or only changes (ChangesOnly=true).  In most cases you can always leave this as "true" and get what you probably want.

The POST result is the Notification which you will use to retrieve the actual data.

1. Read the NotificationItems.  Issue a GET to Notifications/{ID}/Items.

The ID supplied is the ID of the Notification returned in step 2.

The GET results are the NotificationItems you requested.  Paging is expected for the API.

Let me know if you have any questions about this process!

Best Regards,

-Jeff

[mayumi_UWL]

Hi Jeff,

Thank you for your reply and sorry it took so long to get back to you.

Had some back and forth with the customer regarding this issue.

 Before getting into that let me reply to your question first.

1. How many different points are you pulling data from EBO? 246 Inputs/117 Outputs (we are adding more)
2. How often are you pulling the data from these points? - Every minute
3. What data are you using from each of these requests? Values
4. What REST gateway endpoints are you using to get this data? (I see likely you are using /Values/{id}/Value) - Correct

Here are some questions from our customer...

1. Known issue with EBO that if you send too many requests it stops

Q: Is there a roadmap to fixing EBO?

2. Using /Subscription and /Notification instead of GET /Values/{id}/Value

Q: If you are unable to determine that the issue we have is related to the known EBO issue, then what guarantee do we have that this refactoring workaround would resolve the issue?

3.  If your REST Endpoint is using the ‘http’ and not ‘https’ EWS endpoint for EBO, I would suggest changing this to https.

Q: Could we not just turn off the default ‘redirect web clients to https’?
How do we identify if this redirect from HTTP to HTTPS is happening? Is it not possible to see this or the 400 error in the web server log or similar?
Could we not configure the web server to expire/close sessions faster in order to release server resources?

Sorry, if some of the questions are not easy to answer.

Thank you very much for your support.

Regards,

Mayumi