How to keep Processor alive ?

[gllonch]

Hi,

We have an issue in our ModuleExtension (SmartConnector module).

In our code, we are connecting to AlarmItemReader in order to read alarms every 1s, and if a new Alarm appears it will be sent to our software through network by WCF protocol.

The main part of our code is:

protected override IEnumerable<Prompt> Execute_Subclass()

        {

            Logger.LogInfo(LogCategory.Processor, this.Name, $"->Execute_Subclass");

            var executePrompts = new List<Prompt>();

            try

            {

                //Set AlarmItemReader with custom parameters from Web interface of SmartConnector

                AlarmItemReader alarmItemReader = new AlarmItemReader();

                alarmItemReader.Address = EwsAddress;

                alarmItemReader.UserName = UserName;

                alarmItemReader.Password = Password;

                //Start WCF service to expose a Web service though the network which will be used by our software to receive alarm notifications

                StartHost();

                do

                {

                    try

                    {

                        //Wait 1s before reading again alarms

                        Thread.Sleep(TimeSpan.FromSeconds(1));

                        //Check if new alarms are appeared and notify our software in case of new alarm by WCF service

                        CheckAlarms(alarmItemReader);

                        CheckCancellationToken();

                    }

                    catch (Exception ex)

                    {

                        Logger.LogError(LogCategory.Processor, this.Name, ex.ToString());

                        executePrompts.Add(ex.ToPrompt());

                    }

                } while (!IsCancellationRequested);

                //Stop WCF service

                StopHost();

            }

            catch (Exception ex)

            {

                Logger.LogError(LogCategory.Processor, this.Name, ex.ToString());

                executePrompts.Add(ex.ToPrompt());

            }

            Logger.LogInfo(LogCategory.Processor, this.Name, $"<-Execute_Subclass {nameof(executePrompts)}={executePrompts}");

            return executePrompts;

        }

private void CheckAlarms(AlarmItemReader alarmItemReader)

        {

            var lastUpdateCacheKey = $"{this.ConfigurationId}#AlarmItemReader#LastUpdate";

            var result = alarmItemReader.ReadData();

            _cache.AddOrUpdateItem(alarmItemReader.LastUpdate, lastUpdateCacheKey, CacheTenantId);

            if (result.Success)

            {

                if (result.DataRead.Count > 0)

                {

                    foreach (var alarmEvent in result.DataRead)

                    {

                        try

                        {

                            Logger.LogTrace(LogCategory.Processor, this.Name, alarmEvent.ToJSON());

                            //Convert AlarmResultItem to Alarm object of our software

                            Alarm alarm = ConvertAlarmResultItemToAlertAlarm(alarmEvent);

                            if (alarm != null)

                            {

                               //Notify our software of new alarm by WCF service

                                NotifyAlert(alarm);

                            }

                        }

                        catch (Exception ex)

                        {

                            Logger.LogError(LogCategory.Processor, this.Name, ex.ToString());

                        }

                    }

                }

            }

            else

            {

                Logger.LogDebug(LogCategory.Processor, $"Alarm read failed.");

            }

            _cache.AddOrUpdateItem(string.Empty, lastUpdateCacheKey, CacheTenantId);

        }

But sometime, we have an error from AlarmItemReader which is changing the variable “IsCancellationRequested" to “true” and then ModuleExtension is stopped.

So our software cannot connect to the WCF service exposed by this module because it has been stopped by the call of “StopHost” method after receiving this error in “CheckAlarms” method:

2020-03-21 23:13:14.0467,Error,Service,<no principal>,An error was reported while committing a database transaction but it could not be determined whether the transaction succeeded or failed on the database server. See the inner exception and http://go.microsoft.com/fwlink/?LinkId=313468 for more information.

Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

The wait operation timed out

   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext](TTarget target, Action`2 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)

   at System.Data.Entity.Infrastructure.Interception.DbTransactionDispatcher.Commit(DbTransaction transaction, DbInterceptionContext interceptionContext)

   at System.Data.Entity.Core.EntityClient.EntityTransaction.Commit()

   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)

   at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesToStore(SaveOptions options, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction)

   at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)

   at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesInternal(SaveOptions options, Boolean executeInExistingTransaction)

   at System.Data.Entity.Internal.InternalContext.SaveChanges()

   at Mongoose.Service.Data.MongooseDbContext.SaveChanges()

  at Mongoose.Service.Licensing.AntiRollbackDateTime.get_UtcNow()

,["Current WindowsIdentity is NT AUTHORITY\\SYSTEM"],

2020-03-21 23:13:44.0930,Error,Service,<no principal>,An error occurred while executing the command definition. See the inner exception for details.

Invalid operation. The connection is closed.

   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)

   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)

   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()

   at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)

   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()

   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()

   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)

   at Mongoose.Service.Mongoose.HasValidFrameworkLicense()

,[],

Can you tell me if our code seems good to work has expected or if we need to change something ?

Is is possible to “Auto” restart the method “Execute_Subclass" after an error ?

[JeffBowman]

Hello!

By default, a processor will only run for 300 seconds before it is cancelled by the worker manager., this is a safety measure to make sure processors that are supposed to have a finite execution don't get stuck. If you would like a processor to run forever, simply add the interface ILongRunningProcess to your processor class and it won't be cancelled anymore.

Example definition of a long running processor.

public class LongRunningProcessor : Processor, ILongRunningProcess

{}

Best Regards,

-Jeff

[gllonch]

Thank for the feedback but I already implement ILongRunningProcess.

The issue is coming from the exception which is closing my loop because the cancellation token is set to "True" and the method "Execute_Subclass" is exit.

Regards,

[JeffBowman · Accepted solution]

Oh! I see, sorry for the confusion.

The AlarmItemReader code has no way to set the CancellationToken, so it is unlikely from there. From your log file, it looks as though your database is having an issue, and it seems to lose connection to the Smart Conncetor service. Is your database remote or local to the Smart Connector installation?

While I don't know what is causing this DB failure, I can suggest a way to restart the processor automatically.

Create a schedule (Setup -> Configuration Schedules) in your Smart Connector portal and set it to run every 1 second. Then assign this schedule to your processor configuration. This will allow the processor (hence the Execute_Subclass method) to try and run again 1 second after it fails. It sort of acts as a watchdog in this scenario. 

Best Regards,

-Jeff