Geo SCADA MQTT Topic with nested data

[ash_aqx]

Hi All,

I'm wondering whether the MQTT driver is capable of supporting data in nested JSON elements.

The data that I'm receiving is split up a bit differently than what is described in the help files, with timestamps separated from data in the following fashion:

{

"payload_fields":{

    "sensor1":67593

    "sensor2":83463

    },

"metadata":{

    "time":"2021-02-10T12:28:16.002839378Z"

    },

}

I've tried referencing the tag names like this:

• payload_fields.sensor1
• payload_fields.sensor2

But that doesn't appear to be working either. My guess is that because the "time" tag is in a different place to the data GeoSCADA is getting confused..

Any assistance / pointers would be greatly appreciated!

[geoffpatton]

I was only testing MQTT from a Node-Red application, but if I did not have it formatted from Node-Red exactly as documented I got nothing in ClearSCADA. So I think you are not going to get it to work with your current format.

@sbeadlecan probably confirm.

[ash_aqx]

Thanks for the reply,

I thought this might be the case.

Do you know who might be a good contact to get driver enhancements made?

It seems like this is a relatively common complaint from people using the MQTT driver.

It is a touch rediculous that we would need to use an intermediary processor to handle the limitations of the GeoSCADA driver. The entire point of having a SCADA system is that it does these things natively.

[geoffpatton]

From What I understand this is a first go at MQTT and they are going to enhance it. the problem with MQTT is that the messages can be structured any way you want but designing a can driver that is that flexible and still works otherwise with the software is a lot of development work. I am not sure they are ever going to go to that level.

I think they are going to add the the SparkPlug Payload B standard.

Contact your support channel and you can ask them to put in a driver enhancement request.

Steve Beadle who I tagged in my last message might have some information he can share on this.

[AdamWoodland · Accepted solution]

The trouble(*) is MQTT is really just a transport mechanism like TCP. It is like saying we have a TCP telemetry driver when you're interested in DNP3 but can also consider Kingfisher, Modbus, etc, as its just packaged up differently but it is all the same TCP transport (ignoring choosing serial or UDP options!)

Most MQTT communications uses some form of JSON, but I also have to deal with CSV and raw text data and that's before I have to worry about how the data is encoded (e.g. base64) and then what the values mean (e.g. raw readings versus changes since previous reading). Until the various MQTT publishers use standards such as WITS-IoT and SparkplugB dominate, I've yet to interface with a device that uses either, then it is effectively a wild west.

As to how you can solve the problem... you can raise a support request for an enhancement, or...

You can compile the SparkplugB driver from https://github.com/GeoSCADA, Steve has made it open source. I believe there is a WITS-IoT driver around from Steve too, but not sure of the commercial arrangements of that. You could copy Steve's SparkplugB driver and make the relevant changes, its in C# using the Geo SCADA DDK.

Beyond that I do develop our own set of MQTT drivers to interface with the numerous other devices that my customers need to interface with and so its possible you could do the same. or chat to your Schneider rep and they can reach out to me to discuss further depending on which region you're in, or my colleagues elsewhere in the world. Some of our SIs could also help develop a driver if you already have existing agreements with them.

(*) I use the term trouble here, but it is also an advantage with significant flexibility which is one reason why MQTT is so popular in IoT

[ash_aqx]

Thanks Adam,

That's some great insight. Definitely appreciate the complexities. 

Maybe a suite of seperate modules (parsers for JSON, CSV, etc that aren't necessarily tied to the MQTT driver) that can accept and process the output from the MQTT driver might be a simpler approach. Then the user could "build" an appropriate parsrer to suite the application.

Is it possible to intercept the output from the driver and use GeoSCADA scripting to process the payload?

I'll have a look at the driver source that you've linked, might be something there that I can work with.

Again, thanks for the response, very helpful!

[AdamWoodland]

I would love there to be a generic parser but it is beyond my software capabilities (I'm not a product developer, don't worry! 🙂 )

I did do something similar to the interception for some events that came in for a specific device, but in that case just shunted the event details to a structured text program as string/DT and the end user to do with the details what they wanted. The problem is there is no native JSON handling in the easily available server-side languages (i.e. structured text) so you don't really have anything to process the JSON unless you write your own .NET/Python/whatever code and then you're almost back at just putting it in the driver.

[BevanWeiss]

I did look into the possibility of a generic parser a few years ago, and it seemed like it would be reasonably doable actually.

Essentially what it would need is to use some kind of grammar descriptor language, like ENBF

https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form

The issue then being that it's a bit harder for end-users / integrators to 'develop' because such a descriptor language is often quite disconnected from the language that it is being used to describe (i.e. if using it to describe HTML, it wouldn't look very natural to an HTML 'coder').

The real benefit is that it would provide a common platform for AVEVA/Schneider to develop more user friendly interfaces.

The way I'd envisage it would be there would be a drop down of 'common' formats (like xx separated values, or SparkPlugB, etc etc) but then there would also be a 'Custom Format' which would allow the raw ENBF form to be entered / edited.  Ideally the common formats would be implemented in ENBF also, so when transitioning from a common format to 'custom' the ENBF for the common format is shown... hence if just a slight tweak is required to a common format, the end user doesn't have to rewrite it from scratch, they just make the tweak required.

There's example parser generators around, like:

https://github.com/cwbaker/lalr   (does a modified BNF, rather than EBNF)

It would still only support 'character based' protocols, and I'm sure there would be other issues in getting the character data in the first place, but it at least makes the processing of the message data consistent and flexible.

[ash_aqx]

Definitely,

A generic string parser of some description would open up a whole range of lesser used protocols and formats for integration.

Something like this would be super convenient and allow developers / integrators a much shorter path for IoT devices.

[ash_aqx]

Hi Everybody,

Thanks for your guidance and feedback on this topic point.

After the suggestion from @AdamWoodland I cloned and built the Driver-BasicMQTTClient from this repo https://github.com/GeoSCADA.

The good news is that this works perfectly with The Things Stack, providing your region has uplink fields enabled. I used the following topic descriptor: "my-app-id/devices/my-dev-id/up/<field>"

You can check if your region supports it by checking out this link: https://status.thethings.network/