ION7650 Data Recorder Changes Without Explanation

[Scott_MCN]

Hello,

Customer has set up ION7650 to record voltage, current, and power values at 2 second intervals. 

The data recorder unexpectedly stopped recording current and power, and changed to 5-minute recording intervals. 

Attached screenshots illustrate programming, data log from recorder, and event log of meter. 

Can see in the event log that there are no memory failure events, or changes to the modules indicating a change in the timer which triggers the data recorder. 

Has anyone seen this issue before, or know what could be causing the change in interval and other inputs to stop recording?

Table with measurement headers for reference

Shows point in time when 2-second interval changed to 5-minutes

Data Recorder Sources

Data Recorder Output Registers

Data Recorder Setup Registers

Meter Event Log

Timer Setup Registers

[DanL]

Is it possible that you left ION Designer running so that entries are missing from the PME database? The PME Log inserter service is often blocked from retrieving meter logs while ION Designer is running.

Please use ION Setup to directly read that meter's Event log (screen capture shows no events for today) and Datarecorder (depending on which DRE# you are using, you may need to create a "custom table" configuration in ION Setup).

Also please include the "meter config report" from ION Setup to show meter firmware, serial number, template name, configuration options, etc.

I'm also confused/concerned about the "blank" entries in the PME database for current and power. When you shift+click the DRE inputs, are there numeric values for all inputs or just voltages?

[Scott_MCN]

Sir, Lieutenant Dan, Sir! 😁

Meter is connected via Cell Modem – customer only connected briefly to change programming so Designer couldn't have been left on. 

I'll get the event log, data log, and meter config report from ION Setup, and check the input values. 

Data Recorder was created to measure these inputs but otherwise meter programming is original to our knowledge – DRE# is the specific data recorder module used, right? If my assumption is correct, this is Data Recorder #41. 

Another thing to note: memory is 97% allocated including this programming, could near-full capacity be causing this?

Thanks for the quick response,

– Scott

[Charles_Murison]

Could the device have had a framework uploaded to it? ION Setup by default will clear the eventlog after loading a dcf file which could change the programming of the device and not leave a trace in the event log inside PME.

Logging 2 second data over Cell modem, have they calculated expected data rates with this config?

[Scott_MCN]

@DanL  

Responses from the customer:

Q: Is it possible that left ION Designer was left running so that entries are missing from the PME database? The PME Log inserter service is often blocked from retrieving meter logs while ION Designer is running. 

A: We actually used ION setup to create this Data Recorder. Also it was recording fine and just stopped working on it’s own as you saw. Not only that, it started recording only 5 minutes.

Q: Please use ION Setup to directly read that meter's Event log (screen capture shows no events for today) and Datarecorder (depending on which DRE# you are using, you may need to create a "custom table" configuration in ION Setup).

A: Event log from ION Setup Attached. Changes were done yesterday Jan 22nd.

Q: Also please include the "meter config report" from ION Setup to show meter firmware, serial number, template name, configuration options, etc.

A: Meter Config report attached.

Q: I'm also confused/concerned about the "blank" entries in the PME database for current and power. When you shift+click the DRE inputs, are there numeric values for all inputs or just voltages?

A: Yes, when I shift click the DRE inputs, there are values for all 8 inputs that show up. I also see the count on the Period Timer Trigger output increasing about every 2 seconds. The DRE output Record Complete also has a count that’s increasing every 2 seconds.

@Charles_Murison 

Q: Could the device have had a framework uploaded to it? ION Setup by default will clear the eventlog after loading a dcf file which could change the programming of the device and not leave a trace in the event log inside PME.

A: Only our group and load research group have meter passwords to make changes. Telecom would have also had to open comms access to cell modem for someone else to change dcf.

Q: Logging 2 second data over Cell modem, have they calculated expected data rates with this config?

A: We were going to connect to this meter twice a day (12hr period) to obtain a complete 72hr snapshot of this meter to get a high resolution picture. I may change depth to 4hrs to create more available memory. If that doesn't work I may move programming off meter to VIP and have the computer permanently connect to the meter, which we didn't want to start doing. 

Let me know if any of the above gives insight into the data recorder issue. Thanks again for your help!

– Scott

[Scott_MCN]

@DanL 

Event Log too large to attach, here's a DropBox link: https://www.dropbox.com/s/bqi1fwqoddb55ny/Event%20Log%20for%20PJ-1401A039-04.csv?dl=0

[DanL]

You have configured DRE#6 to have a depth of 50400 (28 hours at 2s per log). I'm not sure why you think it is DRE#41?

On every ION7650 I've looked at, DRE#6 is configured by default to log transient module data.

Please also show a screen cap of shift+click on the DRE#6 inputs. I'm curious if the re-arm input is being pulsed and the enable status, etc

The customer meter is an ION7650v371e (same as shipped) with a template version 7650_FAC-4WYE_V1.2.0.1.0 (different from shipped!)

The only template I can find on record with the same version string is 7600_v210_FAC-4WYE_V1.2.0.1.0 (from 2005).

Perhaps Manitoba Hydro took this 15 year old template and heavily modified it to work on the ION7650?

If this is a heavily customized framework that is nothing like the default ION7650 framework, you will likely be required to also customize ION Setup and PME for a consistent user experience.

Although you may have used ION Setup to create/configure DRE6, that is not how you read the logs from the meter with ION Setup. If you have a default installation of ION Setup, it will assume DRE6 is the transient data recorder for 7650 devices, so you should be able to simply view the Data Recorder called "Transients" from the "Data Screens" view.

1. Exit the Setup Assistant window if active
2. Under View menu, select "Data Screens"
3. Open the folder called "Data Recorders"
4. Open the table called "Transients" (this looks at DRE#6 by default for ION7650 meters)
5. The ION Setup default for DRE6 (transient table) is to show the most recent 100 records. Of course you can change your configuration to show more but it will be very slow over a modem

[DanL]

I just thought of a likely reason for the 5-minute interval of "Voltage A-N", "Voltage B-N", and "Voltage C-N" in the PME database query. Since PME stores all data based on the quantity label, it is likely there is another DRE module in that meter that is logging these three quantities every 5 mins (but not the other quantities you are looking for).

Using ION Setup to read the "Transient table" and also the DRE module configuration (inputs, setups and outputs) will aid in determining root cause. ION Setup does not use the PME database. I believe that new logs from DRE#6 are not making it into the PME database. ION Setup will tell us if there are logs waiting to be read in the meter and what the module is doing (shift+click inputs and outputs).

[Charles_Murison]

Hello Scott,

To expand on Dan's comments, can you look at the diagnostic viewer for this device, in particular the Loginserter service section. Some of these tabs will show if there are outstanding records for a device and for a particular data recorder.

Charles

[Scott_MCN · Accepted solution]

Hi Guys,

I'll talk to them about getting the latest templates and adapting them to their standards for existing 7650s. I didn't realize this was a station meter and not a customer check meter with 40 data recorders by default, hence the new DRE assumed as #41 (typically the contact I'm working with is exclusively helping their customers with meters/PME, my bad...)

I've asked the customer to check the ION Setup tables for the data and Diagnostics Viewer for any outstanding records.

Since the last message, he has deleted and reprogrammed the recorder, which seems to have fixed the recording – however, they're checking it in 6-hour intervals to ensure recording, since it stopped before.