NoSQL Database Schema and Interaction

Iotellect Server database schema is fully dynamic, new database entries are created and deleted by the server on the fly in order to reflect system resource creation, removal and reconfiguration.

Different NoSQL formats use different terminology to describe groups of individual entries. For example, key-value databases organize key-value pairs into namespaces or buckets. The analogous structure in Cassandra (a type of wide-column data store) is the keyspace, and document databases use the term “collection”. The term “collection” in the following article refers to this type of higher level structure, but ultimately depends on the type of database being used. Similarly, different databases use different terminology for the list of key-value pairs which exist in an entry. The term “fields” is used in the following article to refer to the list of key-value pairs which occur in each entry of a “collection”.

The basic collections that exist in every installation are as follows:

Collection	Description
context_directory	Contains UUIDs of contexts, used for mapping and extracting data from other collections. Fields: key - full path of context column1 - UUID of context, generated automatically value - creation time in milliseconds encoded in BLOB data type
ag_properties	Contains values of all persistent server context variables. Iotellect Server creates and deletes a relatively low number of entries in this collection when system objects are created and destroyed. Server-side memory caching also reduce the number of queries to this collection. However, entries in the ag_properties collection are extensively updated because values of persistent resource properties are modified and the server saves changed values to the database. Fields: key - UUID of context whose variable's value is stored in the entry property - the name of the variable value - a data table representing the variable value encoded into string using invisible separators
		For example, devices that use persistent value cache cause a lot of ag_properties updates during synchronization.
ag_events	Contains all server context events except for ones that are stored in other (custom) collections. See Context Event Storage section below for details. Fields: key - has a format of UUID:name. Where the first part is the UUID of the context that the event was fired in, and the second part is the name of the event (i.e. event type) column1 - time-based UUID, contains an encoded timestamp of the event occurrence and event ID value - encoded event information, contains: expirationtime - future timestamp of the planned event expiration time level - event level permissions - custom permissions required to access the event instance or null if permissions specified by the event definition should be used count - number of registered event's duplicates ack - an encoded list of event's acknowledgements enrichments - an encoded list of event's enrichments format - an encoded format of the event or null if the format specified by the event definition should be used data - a data table representing event data encoded into a byte array
ag_data	Contains binary data blocks that are embedded into context variable values. Those blocks are extracted into this separate collection when a variable value is being persisted in the ag_properties collection. However, binary blocks are not loaded once the variable value is loaded from ag_properties. The blocks are only loaded on demand, i.e. by explicit requests from Iotellect Server modules, clients or external applications. Fields: key - UUID of context whose data is stored in the collection entry value - binary data

A large Iotellect installation can store billions of persistent context events in the database. Those events are dynamically distributed between different collections in order to optimize storage and retrieval performance.

Format of all event collections matches the format of the ag_events collection described above.

Access to event collections normally comprises a lot of operations to create entries. A new entry is created every time a persistent server event is registered. The entries are deleted from the event collection in two cases:

• If a system resource with all associated events is deleted

• If expired events are purged by a periodic scheduler task

The number of queries performed against an event collection is relatively low. The events are loaded in the following cases:

• If an Event Log is opened, causing subsequent activation of an event filter

• If events are directly loaded by get() function of the Events context

• If variable history is explicitly loaded by variableHistory() function of the Utilities context

• During initial visualization of a widget chart that is configured to include historical events or variable values

• In several similar cases

Despite the fact that events are rarely loaded, the number of events loaded by a single query is not limited by the platform since certain installations require millions of events to be loaded at once. System architects designing Iotellect-based solutions should take care of this and consider maximum number of loaded events when building server-side data processing pipelines and visualization dashboards. An attempt to load an excessively large number of events at once will cause Iotellect Server memory depletion and manifest itself as severe performance degradation caused by Java Virtual Machine garbage collection, possibly causing a full server freeze or internal memory errors.

The event collections list available in this documentation is not complete. Diverse Iotellect Server modules and plugins may request creation of other dedicated event collections that are not mentioned here.

All events that weren't explicitly configured to use a custom collection are written to the ag_events collection. The below list describes database collections that are normally created to store events of different types.

The fields of the entries in these collections are similar to the fields of entries in the ag_events collection.

Direct access of third-party applications to Iotellect Server database is strongly discouraged. Iotellect is a powerful platform offering a mature SDK and diverse kinds of APIs for local and remote access to all entries contained in the server database. Moreover, API-based access will always inherit proper locking, permission checking, and performance optimizations.

Direct modification of data contained in Iotellect Server database will in most cases cause incorrect system behavior. However, direct read operations may be used in rare cases.

Please contact Iotellect for consulting on how to avoid direct database access.

Was this page helpful?

Tell us more

Send

Thank you!