Help
  • Get started
  • Ask the Community
  • How-To & Best Practices
  • Contact Support
Login / Register
Brand Logo
Help
  • Get started
  • Ask the Community
  • How-To & Best Practices
  • Contact Support
close
  • Community Home
  • Forums
    • By Topic
    • By Topic
      EcoStruxure Building
      • Field Devices Forum
      • SmartConnector Forum
      EcoStruxure Power & Grid
      • Gateways and Energy Servers
      • Metering & Power Quality
      APC UPS, Critical Power, Cooling and Racks
      • APC UPS Data Center & Enterprise Solutions Forum
      • APC UPS for Home and Office Forum
      EcoStruxure IT
      • EcoStruxure IT forum
      • EcoStruxure IT™ Advisor CFD
      Remote Operations
      • EcoStruxure Geo SCADA Expert Forum
      • Remote Operations Forum
      Industrial Automation
      • Alliance System Integrators Forum
      • AVEVA Plant SCADA Forum
      • CPG Expert Forum DACH
      • EcoStruxure Automation Expert / IEC 61499 Forum
      • Fabrika ve Makina Otomasyonu Çözümleri
      • Harmony Control Customization Forum
      • Industrial Edge Computing Forum
      • Industry Automation and Control Forum
      • Korea Industrial Automation Forum
      • Machine Automation Forum
      • Modicon PAC Forum
      • PLC Club Indonesia
      Schneider Electric Wiser
      • Schneider Electric Wiser Forum
      Power Distribution IEC
      • Eldistribution & Fastighetsautomation
      • Elektrik Tasarım Dağıtım ve Uygulama Çözümleri
      • Paneelbouw & Energie Distributie
      • Power Distribution and Digital
      • Solutions for Motor Management
      • Specifiers Club ZA Forum
      • Електропроектанти България
      Power Distribution NEMA
      • Power Monitoring and Energy Automation NAM
      Power Distribution Software
      • EcoStruxure Power Design Forum
      • LayoutFAST User Group Forum
      Energy & Sustainability Services
      • Green Building Scoring and Certification Forum
      Light and Room Control
      • SpaceLogic C-Bus Forum
      Solutions for your Business
      • Solutions for your Business Forum
      Support
      • Ask the Community
  • Knowledge Center
    • Building Automation Knowledge Base
    • Remote Operations Devices Knowledge Base
    • Geo SCADA Knowledge Base
    • Industrial Automation How-to videos
    • Digital E-books
    • Success Stories Corner
  • Events & Webinars
    • All Events
    • Innovation Talks
    • Innovation Summit
    • Let's Exchange Series
    • Partner Success
    • Process Automation Talks
    • Technology Partners
  • Ideas
    • EcoStruxure Building
      • EcoStruxure Building Advisor Ideas
      Remote Operations
      • EcoStruxure Geo SCADA Expert Ideas
      • Remote Operations Devices Ideas
      Industrial Automation
      • Modicon Ideas & new features
  • Blogs
    • By Topic
    • By Topic
      EcoStruxure Power & Grid
      • Backstage Access Resources
      EcoStruxure IT
      • EcoStruxure IT™ Advisor CFD
      Remote Operations
      • Remote Operations Blog
      Industrial Automation
      • Industrie du Futur France
      • Industry 4.0 Blog
      Power Distribution NEMA
      • NEMA Power Foundations Blog
      Energy & Sustainability Services
      • Active Energy Management Blog
      Light and Room Control
      • KNX Blog
      Knowledge Center
      • Digital E-books
      • Geo SCADA Knowledge Base
      • Industrial Automation How-to videos
      • Remote Operations Devices Knowledge Base
      • Success Stories Corner
  • companyImpact

Finding large historic granules

Geo SCADA Knowledge Base

Access vast amounts of technical know-how and pro tips from our community of Geo SCADA experts.

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Home
  • Communities
  • Knowledge Center
  • Geo SCADA Knowledge Base
  • Finding large historic granules
Invite a Co-worker
Send a co-worker an invite to the Exchange portal.Just enter their email address and we?ll connect them to register. After joining, they will belong to the same company.
You have entered an invalid email address. Please re-enter the email address.
This co-worker has already been invited to the Exchange portal. Please invite another co-worker.
Please enter email address
Send Invite Cancel
Invitation Sent
Your invitation was sent.Thanks for sharing Exchange with your co-worker.
Send New Invite Close
Top Labels
Top Labels
  • Alphabetical
  • database 32
  • Web Server and Client 31
  • Lists, Events & Alarms 21
  • WebX 19
  • Request Form 18
  • ViewX 15
  • Setup 12
  • Application Programming 12
  • Telemetry 7
  • Mimic Graphics 7
  • Downloads 6
  • Support 5
  • IoT 5
  • Drivers and Communications 4
  • security 4
  • SCADA 4
  • Geo SCADA Expert 4
  • DNP 3 3
  • IEC 61131-3 Logic 3
  • Events & Alarms 2
  • Trends and Historian 2
  • Virtual ViewX 2
  • Lists 2
  • Privacy Policy 1
  • OPC-UA 1
  • Architectures 1
  • Templates and Instances 1
  • ClearSCADA 1
  • Releases 1
  • Maps and GIS 1
  • Tools & Resources 1
  • Mobile 1
  • Geo Scada 1
  • Previous
  • 1 of 4
  • Next
Latest Blog Posts
  • Geo SCADA Expert 2022
  • Geo SCADA 2022 Operating System Support
  • Geo SCADA 2022 Upgrade Strategy
  • Geo SCADA 2022 Upgrade Notes
  • Geo SCADA 2022 Known Issues
Related Products
product field
Schneider Electric
EcoStruxure™ Geo SCADA Expert

Invite a Colleague

Found this content useful? Share it with a Colleague!

Invite a Colleague Invite
sbeadle1
Spock sbeadle1 Spock
Spock
‎2021-06-09 04:01 PM
0 Likes
0
306
  • Bookmark
  • Subscribe
  • Email to a Friend
  • Printer Friendly Page
  • Report Inappropriate Content
‎2021-06-09 04:01 PM

Finding large historic granules

Originally published on Geo SCADA Knowledge Base by sbeadle1 | June 10, 2021 01:01 AM

📖 Home  Back  
ClearSCADA is not designed to handle individual historic granules that are larger than about 1.3MB for Historic Data and 5MB for Alarm and Event Data (see the online help article "Operational Limitations - Server").
Exceeding this limitation may cause extended read locking when the historic granules are loaded, particularly on slower media such as magnetic disks.

Where possible, such granules should be identified and the cause of the abnormally high historic storage rate rectified. It may be advisable to remove the affected granules from the system to prevent performance degradation when searching. An alternative is to alter the stream size, although changing this may require downtime.

The largest granule for each stream can be seen at a glance in the Server Status tool or DBSnapshot in the Historic/Historian page, check the Largest File column.

This article covers a few approaches to identifying large historic granules.

SQL Query


These queries will list the granule sizes and approximate file locations ordered by size including an approximate "% Rated Maximum Size".
The queries select data from the historic index tables CDBHISTORICFILE and CDBEVENTFILE without having to load the granules themselves.

Find large historic granules in the Raw Historic stream.
SELECT TOP( 50 )    RECORDCOUNT AS "_RECORDCOUNT", FORMATVALUE( RECORDCOUNT / 40320.0 * 100.0 USING '0.00' ) || '%' AS PERCENTRATEDMAXSIZE, 'Hd' || 'I' || FORMATVALUE( ( OBJECTID / 256 ) USING '00000' ) || '\' || FORMATVALUE( OBJECTID USING '000000' ) || '\WK' || FORMATVALUE( GRANULE USING '0000000' ) || '.HRD' AS RELATIVEHISTORICPATH, OBJECTID AS OBJECTID, STARTTIME AS PERIODSTARTING, FORMATVALUE( ( ( RECORDCOUNT * 32 ) / 1024.0 ) USING '0' ) || ' KB' AS FILESIZE, CDBPOINT.FULLNAME, CDBPOINT.IDFROM    CDBHISTORICFILE LEFT JOIN CDBPOINT ON CDBHISTORICFILE.OBJECTID = CDBPOINT.IDORDER BY    "_RECORDCOUNT" DESC


Find large historic granules in the Event stream (assuming 256 objects per stream).
SELECT TOP( 50 )    RECORDCOUNT AS "_RECORDCOUNT", FORMATVALUE( RECORDCOUNT / 6825.0 * 100.0 USING '0.00' ) || '%' AS PERCENTRATEDMAXSIZE, 'Jn' || 'I' || FORMATVALUE( STREAM USING '00000' ) || '\H' || FORMATVALUE( GRANULE USING '0000000' ) || '.HRD' AS RELATIVEHISTORICPATH, CAST( ( STREAM * 256 ) AS STRING ) || ' - ' || CAST( ( ( STREAM + 1 ) * 256 - 1 ) AS STRING ) AS OBJECTIDRANGE, STARTTIME AS PERIODSTARTING, FORMATVALUE( ( ( RECORDCOUNT * 768 ) / 1024.0 ) USING '0' ) || ' KB' AS FILESIZEFROM    CDBEVENTFILEORDER BY    "_RECORDCOUNT" DESC



To analyse the distribution of event files per stream:
SELECT  MIN( STREAM ) AS "STREAM No.", COUNT( RECORDCOUNT ) AS "TotalFiles", SUM( RECORDCOUNT ) AS "TotalRecords", SUM( CASE  WHEN RECORDCOUNT > 7500 THEN 1 ELSE 0 END ) AS "7500 plus", SUM( CASE  WHEN RECORDCOUNT > 15000 THEN 1 ELSE 0 END ) AS "15000 plus", SUM( CASE  WHEN RECORDCOUNT > 30000 THEN 1 ELSE 0 END ) AS "30000 plus", SUM( CASE  WHEN RECORDCOUNT > 60000 THEN 1 ELSE 0 END ) AS "60000 plus"FROM CDBEVENTFILEGROUP BY STREAMORDER BY "STREAM No." ASC

To analyse the distribution of event files over time (day base):
SELECT  MIN( FORMATVALUE( STARTTIME USING 'YYYY-MM-dd' ) ) AS "Date", COUNT( RECORDCOUNT ) AS "TotalFiles", SUM( RECORDCOUNT ) AS "TotalRecords", SUM( CASE  WHEN RECORDCOUNT > 7500 THEN 1 ELSE 0 END ) AS ">7500", SUM( CASE  WHEN RECORDCOUNT > 15000 THEN 1 ELSE 0 END ) AS ">15000", SUM( CASE  WHEN RECORDCOUNT > 30000 THEN 1 ELSE 0 END ) AS ">30000", SUM( CASE  WHEN RECORDCOUNT > 60000 THEN 1 ELSE 0 END ) AS ">60000"FROM CDBEVENTFILEGROUP BY FORMATVALUE( STARTTIME USING 'YYYY-MM-dd' )ORDER BY "Date" ASC


Batch Program


The following batch program can be used to find large historic granules.
It will search through a specified directory and all subdirectories for files that are larger than a specific size.

:bof@echo off & setlocal enableextensionsREMREM Recursively finds all files larger than a specified number of bytes within a specified path.REMREM Usage:REM   findlarge.bat "" REMREM Example 1:REM   findlarge.bat "c:\Database\Historic" 2097152REM Example 2:REM   findlarge.bat "c:\Database\Historic" 2097152 > output.txtREMREM:init  for /f "tokens=*" %%f in ('dir "%~1" /b /s /o-d') do (    if %%~zf GEQ %~2 echo %%~zf - %%~ff    )  endlocal & goto :end:end


This is an example of the batch file being run.
C:\utilities\findlarge>findlarge.bat "c:\Users\All Users\Control Microsystems\ClearSCADA\Database\History" 20971525047040 - c:\Users\All Users\ControlMicrosystems\ClearSCADA\Database\History\Historic\HdI000017325184- c:\Users\All Users\ControlMicrosystems\ClearSCADA\Database\History\Historic\HdI00000


"DIR /s"


"DIR /s" can be used to achieve similar results. Use the "DIR /s" command from the root of the historic folders and pipe the output to files. You end up with something like
Directory of C:\ProgramData\Schneider Electric\ClearSCADA\Database\History\Historic\HdI00000\00004003/05/2015  01:44 PM              .03/05/2015  01:44 PM              ..10/08/2014  10:27 AM               160 WK021589.HRD03/05/2015  01:43 PM                32 WK021610.HRD               2 File(s)            192 bytesDirectory of C:\ProgramData\Schneider Electric\ClearSCADA\Database\History\Historic\HdI00000\00004910/08/2014  10:18 AM              .10/08/2014  10:18 AM              ..10/08/2014  10:17 AM                96 WK021589.HRD               1 File(s)             96 bytes


This could be searched to identify large files.

"FORFILES"


"Forfiles" can be used to get a listing from files modified before / after a certain date. Use the "Forfiles" command from the root of the historic folders and pipe the output to files. 
C:\>  forfiles /P "ProgramData\Schneider Electric\ClearSCADA\Database\Journal" /S /D +01/03/2015 /C "cmd /c echo @path,@fsize" > JournalFilesSince0103215.csv

This could be used in case you can't use a SQL query.

Go: Home Back

Author

Biography

sbeadle1

  • Back to Stream
  • Newer Article
  • Older Article
To The Top!

Forums

  • APC UPS Data Center Backup Solutions
  • EcoStruxure IT
  • EcoStruxure Geo SCADA Expert
  • Metering & Power Quality
  • Schneider Electric Wiser

Knowledge Center

Events & webinars

Ideas

Blogs

Get Started

  • Ask the Community
  • Community Guidelines
  • Community User Guide
  • How-To & Best Practice
  • Experts Leaderboard
  • Contact Support
Brand-Logo
Subscribing is a smart move!
You can subscribe to this forum after you log in or create your free account.
Forum-Icon

Create your free account or log in to subscribe to the forum - and gain access to more than 10,000+ support articles along with insights from experts and peers.

Register today for FREE

Register Now

Already have an account? Login

Terms & Conditions Privacy Notice Change your Cookie Settings © 2023 Schneider Electric, Inc