Telemetry

Important

Telemetry is disabled by default. It is a customizable feature which is useful for monitoring Sirveo deployments.

The telemetry feature serves two purposes;

Firstly, as a convenience for us when operating managed deployments in the cloud. Note that cloud-hosted Sirveo and self-hosted Sirveo is exactly the same thing. Furthermore, we operate (“self-host”) deployments for a variety of purposes, including the provisioning of cloud instances. Which is pretty cool. But the point here is that we share the same operational and observability requirements as customers running Sirveo in their own environments. We use default telemetry for cloud-managed deployments.

Secondly, as a feature which allows customers to leverage telemetry for the same purpose we do; monitoring the status of one or more self-hosted deployments.

Configuration

Important

Do not configure SIRVEO_TELEMETRY_MODE to default or both unless you want to enable sharing opt-in telemetry with us.

The following settings are used to configure and enable telemetry to a custom destination;

Setting	default	note
SIRVEO_TELEMETRY_MODE	disabled	accepts disabled, custom, default, both (see below)
SIRVEO_TELEMETRY_URL	none	the destination URL. Required for custom and both
SIRVEO_TELEMTRY_AUTH	none	optional value for bearer token authentication
SIRVEO_TELEMETRY_FREQUENCY	8h	accepts values between 1h and 24h
SIRVEO_TELEMETRY_IDENTIFIER	none	an optional name for the server, for custom mode

• At minimum, set mode to custom, and provide a destination URL.
• Adding an authentication token will add an Authorization: Bearer <token> to requests
• When running multiple servers, add an identifier to distinguish them more easily
• A telemetry event is sent at startup (reason=begin)
• A telemetry event is sent as part of graceful shutdown via SIGINT (reason=end)
• The frequency determines how often in-between events are sent (reason=update)

Modes

custom is for your own telemetry destinations, like your HEC collector, monitoring stack, or maybe a Sirveo webhook

default is for sending telemetry data to our servers. It overrides the other parameters with internal values, which may change between releases. It sends the same data, except that the custom identifier is always excluded.

both will send duplicate telemetry events to your configured destination, and to our servers (using internal parameters).

Important

Using default or both mode always sends events to https://license.sirveo.io, using a fixed 8h frequency that is independent of the configured frequency for a custom destination.

HTTP request details

The server sends telemetry events via HTTP POST requests.

• Data payloads (see below) are sent as JSON, with a Content-Type: application/json header
• User agent is User-Agent: Sirveo
• If configured, Authorization: Bearer <SIRVEO_TELEMTRY_AUTH> is included
• Requests have a 5-second timeout, without retries
• Reponses body content is ignored

The server’s debug-level log output will print response status codes for all modes of telemetry configuration. See below for example log output.

Data payloads

{
    "build_version": "0.1.0",
    "build_commit": "4379644",
    "build_time": "2024-02-16T23:06:23+02:00",
    "build_mode": "licenced",
    "build_os": "windows",
    "build_arch": "amd64",
    "instance_id": "4915a241-ba4e-4...",
    "license_key_hash": "e88394fedd7157...",
    "license_valid": true,
    "reason": "begin | update | license-state-change | end",
    "checksum": "e7916a6b1daa5149",

    // only present for custom destinations, and if configured
    "custom_id": "<your id>"
}

In which:

• The build_* fields are embedded into binaries at build time. See $ sirveo --version.
• The instance_id is a randomly generated UUID which the server generates at startup, and stores in the server_state database table
• The license_key_hash is the SHA256-hashed license key contained within the license (if configured)
• The license_valid field indicates whether the server is satisfied that it has a valid license
• The reason field indicates why the telemetry component decided to send a telemetry payload
• The checksum is directly derived from other values in the payload
• The custom_id is present for custom destinations, and contains the value of SIRVEO_TELEMETRY_IDENTIFIER, or omitted if latter is empty

Specifically excluded from telemetry:

• NO specific details about the OS or system on which the server runs
• NO machine/OS identifiers, beyond a random UUID that server itself generates
• NO data or metadata about feature use
• NO data or metadata about anything that users configure

Note

Telemetry events with license-state-change are sent when the value of license_valid changes.

Instance ID

If it doesn’t exist in a database, or if the existing value is not a valid UUID, a new instance ID is generated at server startup.

A server’s instance ID can be found:

• In the UI (“Server” in left navigation)
• With SIRVEO_LOG_LEVEL=DEBUG, it is printed in server log output at startup
• In the database (select value from server_state where key = 'instance_id')

To change the instance ID of a server for whatever reason, run this query on its database:

update server_state
    set value = ('"' || gen_random_uuid() || '"')::jsonb
where key = 'instance_id'
    returning value as new_instance_id;

Note

Some license types may require a specific instance ID. In which case changing the instance ID will invalidate the license, until the instance ID value is aligned to the license requirement.

Testing a custom destination

An easy way to test telemetry to a custom destination is by sending it to the same (or other) Sirveo server.

• Configure a new graph flow
• Add a single event node
  • name the event telemetry.custom
  • enable “persistent event” to save it
  • enable “include node input” (which is flow input)
• Activate the flow
• Then configure a new webhook to the flow, with a key like inbound-telemetry, and a “None” output mode
• Save the webhook and copy its URL

Then, configure the server like this:

SIRVEO_LOG_LEVEL=DEBUG

...

SIRVEO_TELEMETRY_MODE=custom
SIRVEO_TELEMETRY_URL=<copy the webhook URL>

After restarting the server, you should see this in the log output:

...
INF ready to flow 🟢
DBG [events] <- telemetry.custom id=7229a017 kind=usr persist=true
DBG [telemetry:custom] sent reason=begin status=200

The flow created a new event with the telemetry payload data, which you can inspect via the UI’s event browser.

Tip

Only one telemetry destination can be configured in server configuration. Use a webhook and a task/graph flow to forward (and customize) telemetry to multiple destinations