Azure Event Hub connector

[BevanWeiss]

We've had a number of customers start to utilise the Azure platform for more data analysis, including ingress of larges amounts of IoT data.

This poses a problem of trying to easily connect real time data from the existing OT platforms, like Geo SCADA Expert up into the Azure platform.

I envisage this would be a driver, in a similar fashion to the Wonderware Historian driver currently, where points would be allocated to the Azure Connector, and would then be pushed as a data stream to the configured Azure Event Hub endpoint.

Some amount of rate limiting should be allowed for (to keep the Event Hub load below a configured limit.. which I believe is typically procured in capacity unit allotments where 1 such unit can handle 1MB/s or 1000 events/s whichever occurs first), with a configurable high water mark alarm (e.g. if the connector has buffered > 10MB of data then an alarm occurs) as well as a maximum buffering time alarm (e.g. if the connector has been in a buffering mode for > 2 hours then an alarm occurs).

As a System Integrator we'd like to be able to develop such a connector for our clients, however the Simple DDK doesn't allow for this access to the historical data streams across the whole database, nor does the .NET API.

[BevanWeiss]

I will give a shout-out to @du5tin who already has something around this...

https://github.com/JPISolutions/IoTHubDriver

But not meaning to take anything away from what @du5tin has done, I still think an integrated driver that can directly access the historical data without the overhead of the SQL series, is the better way to handle this connector.

[AdamWoodland]

Hi Bevan,

SE Australia resells a driver developed by SE UK (not the product team) that does most of this called the "High Speed Interface" to customers and SIs. At its basic level allows you to stream historic, events and config out to JSON files on disk using the same on-event function that you mention.

Once on disk as JSON files you can do what you want with it. For our local implementations we write our own service to take those files, transpose to the relevant structure and push to the relevant endpoint. The software also comes with some standard ways to transfer the files off-server that we don't use.

There are some updates planned to the HSI so I don't have any public info available currently. Possibly one of the guys from the UK will see this and chip in with some more info. Get in contact with either you channel manager or the Service Bureau and they can provide some more details if interested or make sure you get details when its available.

(But it is pretty funky)

[BevanWeiss]

Thanks @AdamWoodland, that is useful information.

It does sound a little clunky dropping down to JSON files on-disk to then hoover them up into Azure.

It would be super nice if that "on-event" functionality was exposed to SI's for entire streams.  I've never actually tried to use the .NET SDK to Subscribe to every object in a database... but I suspect it probably wouldn't go well (and I'm sure the .NET Subscription model doesn't account for things like Config changes, or Event Journal data etc).

[sbeadle]

Thank you.