What is the best way to share data between processors?

[JeffBowman]

What is the best way to share data between processors?

[Benji]

You mean threads?

I usually have a central class I pass around as reference

[sesa180908_brid · Accepted solution]

The SmartConnector framework provides two ways for running Processors to exchange data.  Using one over the other is really a matter of preference based on need.

1. ProcessorValue store. Data stored here is actually persisted in the SmartConnector database.  You would obviously use this approach if you needed to ensure that your data would not be lost if the service restarted.  Complex data types are not directly supported but by serializing (XML or JSON) in/outboud of the ProcessorValue store, you could achieve storing structs or reference types just as easily as a primitive value.

1. InMemoryCache. Data stored here is persisted in a thread-safe cache.  References are not persisted across the cache boundary but the InMemoryCache provides generic overloads that allow type safe access to the data.  Data store here is volatile and would be lost with a service restart.  In v2.2 of SmartConnector(due out in September) the InMemoryCache is enhanced to include automatic content expiration and an approach to create logical silos (a cache of caches effectively) where data from one silo is not visible to someone that doesn’t know the cache key.

Both Processor subclasses and Native EWS Server Processor subclasses have access to both of these methods via the base Processor class.  There are also some extension methods available to you to make getting data in and out of them in type safe manner.

[sesa180908_brid]

This approach could work as well, but you would need to safeguard against cross thread access to object reference since each Processor is executing in its own thread.

[Benji]

Yup. Race conditions is a bummer. Till now I only worked with single threads, but being forced to explore mutex 🙂

[Adam_Summers]

Hi Mark,

Yes that's exactly the approach we have been using (mostly just the ProcessorValue store until the recent addition of InMemoryCache), and most recently we used a mixture of the two. Thats's where we wanted to share data across multiple processors and needed this to persist beyond a single run of the SmartConnector service. We stored serialised JSON content to the ProcessorValue store and referenced a flag whilst reading or writing InMemoryCache to block and avoid cross thread issues.

Is there a particular pattern or design which would work better in this case?

[sesa180908_brid]

I don’t think there really is a  right or wrong answer for this.  It all comes down what your needs are.  The database store was implemented if you needed non-volatile storage while implicitly allowing for concurrent reads.  The down side of course is the “last in wins” write approach used by default.  If you needed to ensure only one thing can write (nd not allow a new simultaneous write to overwrite newer content), you would need to implement a semaphore like you did with the InMemoryCache.  I suppose someone could easily write some extension methods that do this as well.

InMemoryCache is faster of course, but has the downside of being volatile and (until 2.2 at least with expiring content support) requires manual cleanup else it would become a memory “leak” of sorts.

I’d be interested in hearing what others think in this regard.

P.S.

In 2.2 I used the InMemoryCache quite extensively in the implementation of the “REST for SBO”. This is what was the impetus for expiring content (and silos) in the IMC - to not cause memory "leaks".