KV Store

Sirveo’s key-value store provides a simple data storage feature. Any flow can access and modify KV store data using the three KV nodes.

The KV store supports the following primary capabilities;

JSON data

The KV store accepts any valid JSON data.

Data caching

Memory-only mode provides an in-memory cache for fast data access or transient storage.

Data persistence

Persistent mode provides database-backed persistance, with automatic and transparent caching.

Data expiry

Optional key expiry between 1 second and 24 hours is supported for both in-memory and persistent KV store data.

Prefixes & keys

When adding an item to the KV store, a prefix and a key is required as a unique identifier for the data. The prefix is technically part of the key, but it is configured separately. Each prefix value provides a unique namespace for keys. This is a simple means of achieving logical separation between different sets of data that flows may store in the KV store.

Tip

To create a unique keyspace for a particular flow, it’s flow ID can be used as a prefix. Find it under the “Details” tab in the flow editor.

Store browser

Persistent store items can be inspected (and deleted) via the store browser in the UI.

Node: KV Set

This node sets or replaces an item in the KV store.

Good To Know

If enabled, persistent items are set in the database before being cached in memory. This behavior ensures that persistent items are not added to the in-memory cache if they can not be saved in the database first.

Basic options

The four basic options configures the key, prefix, persistance and an optional expiry period.

Key is required, and may be a dynamic value, like ${data:some-value}.

Key prefix defaults to global if not provided. Custom prefixes can be used to isolate different datasets from each other.

Good To Know

Prefix values may not be dynamic. This is a security-related constraint, to prevent configurations which allow external flow input data - which may be untrusted - from determining where KV store items may be read from and written to.

Persistent determines whether an item is persisted in the database, or only in memory.

Warning

Store items that are set without persistence enabled only exists in server memory for up to 24 hours, or until the server process stops or crashes.

Secondly, the KV Set node cannot be used to make existing persistent items non-persistent. However, it can make non-persistent items persistent. It can also add an expiry time to non-expiring items, whether persistent or not. This behavior is a feature, not a bug.

Expiry is an optional expiry time for the store item. If provided, the expiry period applies to both persistent and non-persistent store items.

Good To Know

Non-persistent items will only exist in the server’s in-memory cache, for up to 24 hours. To permanently store an item, enable the persistent option, and do not configure an expiry period.

Input options

When setting a key value, that value may either be obtained as-is from the node input, or configured manually as literal JSON data.

Tip

Variable expressions can be used in literal JSON.

Output options

By default, this node will output the following JSON:

{
  "ok": true,
  "value": {
    "some": "data"
  }
}

ok is true if the item was successfully set. This node may fail to set store items in some scenarios:

• A dynamic key (e.g. ${data:key}) results in an invalid key value
• The input data is not valid JSON
• The input data is too large
• If a persistent item could not be stored in the database (e.g. connectivity issues)

If ok is not true, an error property will be present in the output, to describe the problem.

value contains the stored JSON data.

As a convenience, the node can be configured to automatically add the values of ok and value into the flow’s $data store, using the provided values as property/key names.

The last two options are used to include the key and prefix value in the node output, as _key and _prefix properties, respectively.

{
  "_key": "my-key",
  "_prefix": "my-prefix",
  "ok": true,
  "value": {
    "some": "data"
  }
}

Node: KV Get

This node retrieves a KV store item by prefix and key.

Basic options

Read Mode controls how the value is read. Options are:

• Cache then database
• Database only (ignore cache)
• Cache only (ignore database)

Good To Know

If the Read Mode option results in an existing, non-expired key being retrieved from the database, that key is automatically cached in memory. When automatic caching occurs, the cached item will include the key’s expiry time, if configured.

Default JSON is optional, and used as the output value when the item is not found.

Other options have the same meaning as the options in the KV Set node.

Output

If a key is found, this node’s output looks like this:

{
  "ok": true,
  "value": {
    "some": "data"
  }
}

The ok property indicates whether the item was found or not.

Non-existent or expired keys will contain a value of null.

{
  "ok": false,
  "value": null
}

Unless a default value is configured:

{
  "ok": false,
  "value": {
    "some": "default"
  }
}

An error property may be present, which will describe why the read operation failed. Read operations may fail when:

• the server can’t reach it’s databases (if Read Mode involves database retrieval)
• an invalid key value, which may happen with dynamic key values (e.g. ${data:some-key})

Node: KV Keys

This node lists up to 1000 keys for a given KV store prefix.

Good To Know

When Read Mode is Cache then database, keys are aggregated from the cache and then the database, up to the configured or maximum limit.

When Read Mode is Cache only (ignore database), the sort order of keys is not guaranteed to be stable.

Basic options

These options are generally the same as in the KV Get node, with some additions.

Output

When the Output keys directly is disabled, output looks likes this, assuming 3 keys are found for the prefix:

{
  "keys": ["a", "b", "c"],
  "ok": true
}

When enabled, the node output only contains an array of keys, which may be empty if no keys are found, or if an error occurs.

["a", "b", "c"]

Node: KV Remove

This node attempts to remove the given key from both the in-memory cache and the database. Options have the same meanings as the KV Set node.

Output

Absent configuration issues or database connectivity problems, node output is:

{
  "ok": true
}

The value of ok is true if the remove operation was successful, and acts as confirmation that the target key is no longer present in the KV store. It is not an indication of whether the target key existed or not, prior to the remove operation.

If ok is not true, an error property will describe the problem, which may be:

• a database connectivity issue
• an invalid key value, which may happen with dynamic key values (e.g. ${data:some-key})

Notes about expiry

1. When a store item is set with an expiry period, that expiry period applies to both persistent and non-persistent store items. Persistent items will not be cached for periods that exceeds the configured expiry value.

2. The expiry option of an existing persistent or non-persistent store item will reflect the configured (or absent) expiry setting of the last KV Set operation.

3. Persistent store items with an expiry option are removed by the server’s database cleanup job, which runs at startup, and every 15 minutes thereafter. As such, they may exist in the database for up to 15 minutes beyond their expiry time. Prior to cleanup, expired store items in the database appear as non-existent to the KV Get node.

4. When a store item is set without database persistence enabled, it exists only in the server’s memory for up to 24 hours, or until the server process stops (or terminates abnormally).

Constraints

These constraints apply to version 0.2.8 and later.

Prefix values
All prefix values must satisfy the regular expression ^[a-z0-9_\-.]{4,40}$. This means between 1 and 40 characters, which may include lowercase letters, numbers, underscores, dashes and periods. The following prefix values are reserved for internal use, and cannot be set, read or removed by KV nodes in flows:

• __int
• __srv
• __flow
• __web

Key values
All key values must satisfy the regular expression ^[a-zA-Z0-9_\-.]{1,100}$. This means between 1 and 100 characters, which may include uppercase and lowercase letters, numbers, underscores, dashes and periods.

Store item values
The KV store will not accept a value if the size of it’s serialized JSON representation exceeds 5120 bytes.

Expiry period limit
The optional expiry period for store items may not exceed 24 hours.

Cache item limit
The in-memory cache currently accommodates up to 20,000 items, to prevent unbounded memory usage. At this limit, the oldest item is removed when a new item is cached.

Note

The KV store’s initial implementation in version 0.2.0 had a 10,000 cache item limit, and a 2048-byte value limit. In version 0.2.8, these constraints relaxed to 20,000 cache items with up to 5120-byte values.