Granulation

The Granulation Engine is purposed for time-series data aggregation. This is the same purpose as for the Statistics subsystem in Iotellect, but the Granulation engine extends beyond the limited set of aggregation options the statistics channels offer. This comes at the price of lower performance and higher memory/disk space consumption.

The most important features of the Granulation Engine are:

• It allows to calculate and store virtually any custom data with any data structure required, not just plain numbers. Of course, this means the standard aggregation functions are also supported. For example, the average function can be implemented by collecting samples count and their sum.

• Unlike statistics channels, the Granulation engine makes it possible to aggregate data over virtually any (custom) periods of time, evenly or unevenly spaced.

• The Granulation engine never 'closes' aggregation periods, making it possible to add information to any period at any moment of time. That, in particular, means data can be processed in any order: if some data arrives with a delay, you still can process it and add aggregate to an interval in the past.

If you are familiar with RRD-based statistics, you can draw an analogy between it and the Granulation Engine. RRD-based statistical data is presented by named Statistics Channels, each one processing the values of a context variable. This is very similar to the Granulation Engine, but the Statistics Channel analog is called Granulator here.

To quickly set up Granulation with a step-by-step wizard, open the context menu for any variable and select Configure Statistics and Granulation.

A Granulator is defined in a context with the following attributes:

Data is accumulated in "granules". Each granule represents a certain time interval, over which the data is aggregated and contains:

• A timestamp that represents a time interval the granule stores information for.

• An accumulated value for the time interval. The accumulated value is a Data Table with format defined in the Value Format field.

A granule's timestamp is supposed to be in a one-to-one relationship with the time interval it presents; at the same time, the timestamp uniquely identifies this granule within its granulator.

For example, a granulator that aggregates average values hourly should have a granule for every hour; those granules can be identified, e.g., by time stamps with zero minutes, seconds, and milliseconds. There are, of course, other ways to implement that; this depends on the developer's choice.

An enabled granulator is automatically activated when the variable it is associated with is changed. Alternatively, the granulation can be invoked by calling the Granulate function in the context the variable is defined within.

The granulator takes the timestamp of the change event and the newly arrived variable value, finds or creates a granule that represents the time interval the event falls in, and updates the value of the granule.

Thus, the granulation algorithm has the following stages:

• evaluating granule timestamp that identifies the required time interval and granule

• acquiring a granule by the calculated timestamp

• updating granule data according to the newly arrived value and storing it

Below, the algorithm is covered in more detail.

First, the granulator must identify the time interval in which the incoming data falls. Intervals are identified by timestamps.

The Timestamp Expression is used to map the event time into a granule timestamp. A developer should ensure the mapping is 1:1.

When the Timestamp Expression is being evaluated, the variable update time is passed into the Resolution Environment as a single-valued default data table. Thus, to get the update time, the following expression can be used:

cell(dt())

This value can be mapped to a granule timestamp. For example, to aggregate minutely stored data, you can use the following expression for granule timestamps:

date(year(cell(dt())), month(cell(dt())), day(cell(dt())), hour(cell(dt())), minute(cell(dt())), 0)

Mind, that you can easily provide unevenly-spaced calendar periods, like months, for example:

date(year(cell(dt())), month(cell(dt())), 0, 0, 0, 0)

Granules are stored as events for the context the granulator is defined in, according to the following rules:

• The event name is the same as granulator's one

• The granule timestamp is kept as the event creation time

• Granule data is stored in event's data table.

With a timestamp, the granulation engine finds a granule that represents data for the corresponding time interval.

If no granule exists yet, a new event is created with the timestamp and data table of the format specified in the Value Format field.

The Value Expression is used to calculate the new granule value based on the incoming variable and previous granule values.

The expression evaluation environment includes:

• the new variable value as a default data table; so, it can be accessed using the dt() function;

• the current granule value the 'previous' environment variable; use {env/previous} to get its value.

The result of the Value Expression evaluation is stored back into the same event as a new value for the granule.

Keep in mind that the new value is being unwrapped from the table before storing its value to a granule. That may cause inconveniences when trying to store a multi-record data table as a granule value. In such cases, do not forget to wrap the resulting value in an outer data table:

array("data", dt())

On granule update, a corresponding event is triggered in the owner context (see below).

This function allows you to explicitly invoke variable granulation. This can be useful, for example, to update a variable retrospectively when data comes with a delay.

The function takes the variable name, new value, and timestamp (as shown in the table below) and starts the granulation algorithm described earlier.

Granules can be read as conventional events in the context the granulator is defined in, e.g., using the get function from the events context.

For example, to get all the granules of a granulator named 'testGranulator' in the 'users.admin.devices.virtual' context, you can use the following expression:

{events:get("users.admin.devices.virtual", "testGranulator", NULL, NULL, NULL)}

When granulation is enabled for a context, the Granule Updated event with the following format is added:

The event is triggered when a granule is updated.

Was this page helpful?

Tell us more

Send

Thank you!