Statistics Channel Properties

This article covers properties of a single statistical channel.

The Expression property defines what data is the base of statistics. It is evaluated on every change of variable that the channel is built upon.

The expression must evaluate to a number (integer of floating point). If the expression evaluates to any other data type, the system tries to convert it to a number and skips the value if conversion fails. If the expression evaluates to NULL, the system applies NULL Value Handling method to it (see below).

A channel may include one or more Datasets for a single context variable. Multiple Datasets can be created for tabular variables. In this case each row in the variable is identified by unique string key (Dataset name). See the Working With Key section for further explanation.

Statistics channel provides several options for post-processing values returned by channel Expression:

• NULL Value Processing

• Out Of Range Values Processing

• Minimal Value

• Maximal Value

This setting defines that should the channel do when its Expression was evaluated to NULL. There are two options:

• Discard. Value will be silently dropped. If no other values were received for a certain period, channel overall value becomes UNDEFINED for this period.

• Treat As Zero. NULL values will be converted to zeros.

Specifies how to process the values that are less than Minimum Value or greater than Maximum Value. The following choices are available:

• Do Nothing. Simply disables minimum/maximum values for the channel.

• Discard. Drops values that are less than minimum or greater than maximum. If no other values were received for a certain period, channel overall value becomes UNDEFINED for this period.

• Normalize. Values less than minimum and greater than maximum are converted to the minimum and maximum values respectively.

These numbers define discarding/normalization thresholds, as specified by Out Of Range Value Processing option.

When enabled for the counting-type channels (Counter, Derive and Absolute), these numbers define minimal and maximal per-second rates.

Minimal and Maximal Value settings are very useful to handle counter resets and erroneous decreases. If you've a counter that can reset to zero or decrease at any time (e.g. during device reboot), Iotellect Server has no way to detect whether the counter has been reset or overflowed. However in case of counter reset or decrease calculated per-second rate will be very high. For example, if counter value measured at 18:00:00 was 100 000 and the next reading at 18:01:00 resulted to 1000, statistics module will suppose that the counter was increased up to 2^32 (4.2 billion), overflowed, and increased to 100 during this minute. Thus, calculated per-second rate will be definitely higher than 1 000 000. To exclude such high rates caused by counter resets from your statistics, just set Maximal Value to something definitely higher than your maximum expected per-second rate but lower than abnormal "counter reset rate". For example, if your maximum expected per-second rate is 10 000, set Maximum Value to 100 000 to throw incorrect rates away and avoid statistics cluttering.

Basically, statistical data may be stored in files and memory. File-based storage is persistent, while data of memory-based channels is discarded when server is restarted.

Storage property defines statistics storage type. There are several options:

• Mapped file (Higher memory usage, faster) - the default setting.

• Standard file (Lower memory usage, slower).

• Memory. Memory based storage guarantees maximum performance, but it doesn't preserve data between server restarts and, therefore, has very limited uses.

• NoSQL Storage. Leverages database-level failover, but is much slower compared to file storage.

It is important that NoSQL Storage only works with NoSQL Event Storage enabled in Database settings.

Here are some guidelines for making a choice between standard files and memory-mapped files:

• Memory-mapped file caches channel data in RAM, while data from standard-files is kept on disk.

• Running memory-mapped file cache will allocate additional RAM from system resources. The additional RAM used by memory-mapped file caches is in addition to the RAM allocated to the JVM heap. Since this memory usage is outside of the JVM heap, it will not be visible in typical Java monitoring dashboards by default. Additional plugins or settings may be required to visualize the additional memory usage.

• The additional RAM consumed by the operating system as a result of using memory-mapped storage will be approximately equal to the size on disk of all statistics files (that is, the total size of the statistics folder).

• Both file-based storage's performance dramatically depends to the operating system, so in most cases it's necessary to perform some tests to select the most suitable storage option.

The channel Type setting defines how are subsequent source values processed to form Primary Data Points for the channel. See Channel Types for details.

Statistics module use Aggregation Functions (AF) to convert Primary Data Points into Consolidated Data Points (see RRD Technology for details). There are size aggregation functions that may be individually enabled/disabled for a channel:

• Average

• Minimum

• Maximum

• Total

• First

• Last

Data aggregation functions are applied to all Archives enabled for the channel. For example, if Average aggregation function is enabled, the system keeps Minutely, Hourly and other averages.

This function calculates an average of Primary Data Points to form a Consolidated Data Point. This is an average of values themselves (for Gauge channel) or value change rates (for Counter and other channel types).

This function takes finds minimal Primary Data Point to use as a Consolidated Data Point. This is minimal value itself (for Gauge channel) or minimal value change rate (for Counter and other channel types).

Similar to Minimum.

This function sums all Primary Data Points to form a Consolidated Data Point. This is a sum of values themselves (for Gauge channel) or value change rates (for Counter and other channel types).

This function takes first Primary Data Point as the value for a Consolidated Data Point.

This function takes last Primary Data Point as the value for a Consolidated Data Point.

Each channel can have all or some of the following Archives enabled:

• Minutely

• Hourly

• Daily (fixed 24 hours period, see note below)

• Weekly

• Monthly (fixed period, see note below)

• Yearly (fixed period, see note below)

Important note: the RRD database and statistics module are always dealing with fixed length time periods. This leads to the following consequences: 1. Daily archives are set up for the 24-hour days. This will produce incorrect daily results for days when switching from DST (summer time) to normal time and back occurs. 2. Monthly archives are set up for an average month length during any four consequent years (three normal years and one leap year), which is around 30.43 days. This affects the precision when calculating per-month and per-year statistical values for all months/years. To get more precise monthly and yearly results, you need to base your statistics on daily samples and use additional aggregation on the later steps of processing, e.g. chart's internal data aggregation that will group daily samples by "real" months/years. To get results with highest possible precision level, base your calculations on hourly statistical samples and use higher-level aggregation to group results by days, weeks, months or years.

Length of each archive, i.e. number of Consolidated Data Points in it, is configurable.

To completely disable an archive set its length to NULL (<Not set>).

System administrators define length of archive in "human" time units, while number of Consolidated Data Points is calculated automatically. For example, if length of Minutely archive is set to 1 day, it will contain 1440 CDPs (i.e. 1440 Averages, 1440 Minimums, 1440 Maximums and 1440 Totals, having all that aggregation functions enabled).

This expression allows to override a timestamp for each sample. The expression should return a Date. Each time a new sample arrives this expression is evaluated to get a sample timestamp. When Timestamp Expression is not set, a current time is considered as a sample timestamp.

This is a utility flag to protect statistics channel from deletion. If one of the following two parameters (Step and XFiles Factor) is changed, the channel is reset and its data is lost. So if you need to change protected parameters you should uncheck Protect Statistics first.

Granularity interval for the channel, see RRD Techology for details. The Step should not be changed in most cases.

Changing Step will reset channel statistics.

Since the default step is 60 seconds, the Average, Minimal and Maximal statistical values for the Last Minute period will be equal.

The XFiles Factor defines what part of a consolidation interval may be made up from unknown Data Points while the consolidated value is still regarded as known. It is given as the ratio of allowed unknown Primary Data Points to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).

Changing XFiles Factor will reset channel statistics.

The default value is 0.9, meaning that 90% of initial samples may be missing.

To ensure more reliable statistical values, decrease this value. The typical setting is 0.5.

Defines whether this channel's brief statistics should be shown in the status of container (i.e. device or tracker).

The maximum time period that may pass between two source values received by the channel before the statistical value for a certain period is assumed to be unknown.

Zero value means automatic heartbeat.

Automatic heartbeat is calculated as follows:

• If statistics channel is bound to one of the remote device settings (which are listed in Device Setting Synchronization Options), heartbeat is calculated as expected synchronization period for the setting (minimum of full and partial synchronization periods) multiplied by 2.

• In all other cases there is no way to automatically determine value updates period so the heartbeat is constantly set to 1 minute multiplied by 2 (120 seconds).

Automatic heartbeat intended to work in statistical channels that are bound to remote device settings.

Heartbeat period should manually be set to twice as long as the expected variable update period of the channel when automatic heartbeat is not applicable. If the heartbeat period is lower than the variable update period, statistics cannot be generated.

Statistical channel receive source values at arbitrary times. From them it builds Primary Data Points (PDPs) at exact times every Step interval. The PDPs are then accumulated into CDPs. See RRD Technology for explanation of what's a PDP and a CDP.

The Heartbeat defines the maximum acceptable interval between samples. If the interval between samples is less than Heartbeat, then an average rate is calculated and applied for that interval. If the interval between samples is longer than Heartbeat, then that entire interval is considered unknown. Note that there are other things that can make a sample interval unknown, such as the rate exceeding limits, or even an unknown (NULL) input sample.

The known rates during a PDP's Step interval are used to calculate an average rate for that PDP. Also, if the total unknown time during the Step interval exceeds the Heartbeat, the entire PDP is marked as unknown. This means that a mixture of known and unknown sample time in a single PDP Step may or may not add up to enough unknown time to exceed Heartbeat and hence mark the whole PDP unknown. So Heartbeat is not only the maximum acceptable interval between samples, but also the maximum acceptable amount of unknown time per PDP (obviously this is only significant if Heartbeat is less than Step).

The Heartbeat can be short (unusual) or long (typical) relative to the Step interval between PDPs. A short Heartbeat means you require multiple samples per PDP, and if you don't get them mark the PDP unknown. A long heartbeat can span multiple Steps, which means it is acceptable to have multiple PDPs calculated from a single sample. An extreme example of this might be a Step of 5 minutes and a Heartbeat of one day, in which case a single sample every day will result in all the PDPs for that entire day period being set to the same average rate.

Was this page helpful?

Tell us more

Send

Thank you!