Disk Usage Optimization

This chapter explains how to optimize Iotellect Server disk space usage.

A highly loaded Iotellect server receives and stores hundreds of thousands data samples per second. These data samples normally flow in as device events or values of device variables which are also converted to events and stored in the database. Reducing amount of disk space used by event history is crucial for high load environments.

There are several methods that can greatly decrease the disk footprint of time series history:

• Switching off storage of non-important event parameters

• Using database-level compression

• Intelligent filtering of value flow

• Limiting history storage time

• Using value aggregation

This option assumes that your mass-produced events are moved to a dedicated Event Storage which is mapped to a separate database table.

Each Event Storage allows fine-grained configuration of stored event parameters, such as event context, name, level, permissions, count of found duplicates, acknowledgements, enrichments, and more.

Disabling persistence of all these parameters will cause only the raw event data (i.e. “payload”) to be written to the disk, significantly reducing disk space usage.

Most database engines support generic database-level or table-level data compression. Since time series history is often a stream of similar or even equal values, such compression may often decrease disk footprint several times, sometimes more than ten times.

For example, Cassandra database offers several compression methods rated by their compression ratio and compression/decompression speed.

Note that compression methods with higher compression ratio tend to have lower compression/decompression speeds and higher CPU load impact.

Smart value stream filtering assumes that some values received from devices are excluded from processing and/or storage. Here are some examples:

• All values with bad or unknown OPC quality can be persistently stored at their full read/receive rate while only every Nth value with good quality can be stored

• If a PLC is measuring grid frequency at 100 millisecond rate, every reading can be stored during the periods when it deviates from its 50/60 Hz reference value for more than a certain percent. Normal frequency values can be stored once per second.

• Newly read/received values can be persistently stored only if their difference from the previous value exceeds a certain absolute or relative threshold. This allows to keep all values once they quickly change (e.g. once a failure is developing) while most values are skipped during the ordinary process flow.

There are two device setting synchronization options that affect persistent value storage:

• History Mode/Rate parameter is an easy way to exclude values with certain quality

• Filter parameter allows to employ advanced expression-based filtering

The History Storage Time parameter of device setting synchronization options controls the historical value retention period. Make sure to configure it so that you’re not preserving the history for longer than it’s required by your project/product requirements.

Values that exceeded their storage time are permanently deleted from the database effectively reducing its size.

For older historical values, it’s often enough to keep only statistically valuable information, such as hourly minimums/maximums or minutely averages.

Use Statistics or Granulation to pre-aggregate values for long-term history storage.

Both of these mechanisms are much more efficient in terms of disk footprint when compared to raw history storage.

You can also combine raw history storage for short-term history and aggregated history storage for long-term history, gradually reducing level of available history details.

Some devices connected to Iotellect Server may produce billions of events or values updates per day, or hundreds of thousands of updates per second. Iotellect Server is capable of processing and even persistently storing such a large number of events/updates. However, some server and device account options should be fine-tuned to reach good performance figures.

• Make sure that Iotellect Server uses NoSQL database for event and configuration storage

• If magnetic (not solid-state) drives are used as a primary historical data storage, tune the NoSQL database settings to configure Commitlog Directory and Storage Directory to be located on different physical disks. To enable high performance, make sure that both commit log and primary storage are located on solid-state drives (SSD).

• Open device account properties dialog and set Settings Cache Mode advanced setting to Memory (transient cache). If the default Database (persistent cache) type is used, every device setting variable update is written to disk twice: as a persistently cached current setting value and as a historical value update event. This significantly decreases overall system performance.

• Most communication protocols designed for high performance allow devices to send asynchronous value updates that doesn't require any polling from the server side. Examples of such protocols are OPC or IEC 60870-5-104. If your device driver supports asynchronous value updates, set Synchronization Period to a larger value (e.g. one day) to prevent non-efficient polling.

Was this page helpful?

Tell us more

Send

Thank you!