Calculate Standard Deviation in an ION Meter

[Sepehr_Zarrabi]

Is it possible to somehow calculate Standard Deviation in an ION meter (ideally PM8000)? We have an opportunity to sell multiple PM8000's to a firm that audits wind turbine production. We can meet everything they need to do except calculating Standard Deviation for power (kW) over 10-minute periods. 

One possible solution is to log kW every second and calculate Standard Deviation in the software but given the power of ION, I am hoping we can come up with a framework that does this calculation on board the meter. 

I would appreciate any suggestions.

Thank you!

[Hal_Etheridge]

Normally to calculate standard deviation you need the complete data set.  This would be difficult to do on the meter itself using that algorithm.

A few years ago I came across a method to incrementally calculate the standard deviation on data as it is being received (ie. a streaming operation).  This can likely be implemented in a ART module with some creative formulae.  I am not sure if accumulated error due to the repeated floating point operations will impact the final value enough to matter though.  Internally the meter ART module uses 32 bit floating point numbers so this may be fine if the number of samples is kept "small"... a 15 minute interval's worth of data will probably meet this requirement but you would need to do some testing to be certain. 🙂

The following may help:

https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance  (See the Welford's online algorithm section)

https://dev.to/nestedsoftware/calculating-standard-deviation-on-streaming-data-253l

[Sepehr_Zarrabi · Accepted solution]

Thank you for your response @Hal_Etheridge. I basically came across the same thing that the Standard Deviation must somehow be calculated in real-time as data is "steaming".

I came across this pseudo code, which I was able to implement in the PM8000 meter using an Averaging module and a couple of Arithmetic modules and Store module (ION framework attached).

double std_dev2(double a[], int n) {
    if(n == 0)
        return 0.0;
    double sum = 0;
    double sq_sum = 0;
    for(int i = 0; i < n; ++i) {
        sum += a[i];
        sq_sum += a[i] * a[i];
    }
    double mean = sum / n;
    double variance = sq_sum / n - mean * mean;
    return sqrt(variance);
}

I ran some tests and the results were very promising:

    Sample set 1: 1 to 900.

        The Std Dev as calculated by Excel is  259.8074607602.

        The value calculated by the PM8000 is 259.8074340820.

        The values match to within 4 decimal points.

    Sample set 2: 1000 to 1600.

        The Std Dev as calculated by Excel is  173.4935157290.

        The value calculated by the PM8000 is 173.4935150146.

        The values match to within 5 decimal points.

I have been around ION technology for 17 years and it still amazes me with how powerful it is!

Thank you for looking into this and responding.

[Tom_Stevenson]

Very cool Sepehr !

It would be cool to post the sub-framework here if there are no hoops to jump through - cool application !

@DanL 

Tom

[DanL · Accepted solution]

I agree and confirmed that using just 4 modules (two AVG and two ART), the standard deviation of up to 16 quantities can be computed in a PM8000 meter.

The first AVG module has "Calc Mode" set to "RMS Mode"

The second AVG module has "Calc Mode" set to "Average Mode"

Both AVG modules have their reset linked to "Rev Intvl Reset" (output of PulseMerge#6 in the FWN I'm looking at)

The ART modules each have their source inputs linked to a group of 8 AVG module outputs.

The formulas in the ART modules will look like "SQRT(S1^2-S2^2)"

Where S1 and S2 are the RMS and AVG of a particular quantity respectively.

The results you should expect will typically be in agreement for the first 7 significant (decimal) digits - which is what to expect since ION meters uses 32-bit floating point (IEEE 754 single precision aka. binary32). Excel uses 15 digits of precision.

The effect of subtractive cancellation can reduce the digits of precision as the standard deviation result approaches zero, but the absolute accuracy will still remain.