Outbound HTTP

The ability to easily and flexibly communicate with other systems and online services via HTTP is an important core feature in Sirveo’s flows. HTTP is also Sirveo’s primary means of outbound integration.

Within flows, outbound HTTP requests are performed with the HTTP client node. It provides relatively low-level customization options with a focus on flexibility. Notable capabilities include:

Various request methods

GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, and TRACE is supported.

Request header customization

Configure arbitrary request headers, except Content-Length.

Dynamic data

With variable expressions, it’s easy to dynamically generate request headers, query parameters and request bodies.

Strict TLS validation

TLS requests to HTTP servers with invalid certificates with fail by default. Support for self-signed certificates is available via explicit, opt-in configuration.

Precise request timeouts

Specify custom request timeouts in milliseconds, between 1 and 20 seconds.

Flexible request body generation

Input data for request bodies data can be specified statically in the node configuration, or dynamically via input from another node.

Automatic JSON support

Since JSON is the “native” data format within flows, the HTTP client node automatically handles JSON request and response data.

Form data support

Making HTTP requests with form data has first-class support, with automatic encoding. This simplifies simulating browser behavior for form submission.

Plain-text support

While JSON and form data has first-class support, the HTTP client node supports arbitrary plain-text request bodies with custom Content-Type headers.

User-defined response statuses

Apart from intuitive defaults, one or more expected response statuses can be configured to control error behavior.

Output control

Output behavior is configurable, including response body data and complete headers.

Response post-processing

Perform complex introspection and/or modification of response data and/or headers with Javascript, with support for overriding error behavior.

Configuring HTTP requests

The HTTP client node can be used in any flow mode. Minimal configuration requires a URL and a request method.

URL
Required. should be a valid URI with an http:// or https:// scheme. Dynamic values are supported via variable expressions.

Method
Required. GET requests may not include a response body. Form data is ignored for GET, HEAD, and OPTION methods.

Request Timeout
The default request timeout is 10 seconds (10000 milliseconds).

Send input data
When enabled, the HTTP client node’s input data will be used as the request body. When no Content-Type header is configured, application/json; charset=utf-8 is used by default.

Security note

Please be mindful when configuring this node to send input from other nodes. See the security reference for important recommendations on using this feature safely.

Emit the request URI
When enabled, the target URI is included in the node output, in a uri property. Useful when troubleshooting configurations with dynamic URLs.

Emit Response Body
When enabled, the response body (if present) is included in the node output, in a body property. JSON data is automatically decoded. Non-JSON data (e.g. HTML, plain text) is encoded as a string.

Emit Response Headers
When enabled, the response headers are included in the node output, in a response_headers property.

Allow Insecure Certificates
When enabled, requests to servers with an invalid (typically self-signed) SSL certificate will not result in an error. In this case, node output will include an insecure property, with a value of true.

Request Headers
Configure custom request headers to include in the HTTP request.

Example

For typical API authentication schemes with a bearer token, add an Authorization header with a value of Bearer ${vars:my-api-token} to retrieve a token from a server variable.

Automatic header behavior

1. The Content-Length request header is handled automatically, and cannot be customized.

2. When form data is configured, Content-Type is automatically set to application/x-www-form-urlencoded. In this case, a Content-Type of multipart/form-data can be used to enable multipart behavior, but other Content-Type values are ignored and overwritten.

3. When a User-Agent request header is not configured, User-Agent: sirveo is automatically added to requests.

Query Parameters
Any query parameters will override existing query parameters which may be configured directly in the URL.

Form data
Add key-value pairs to be used in the request body. URL encoding of values is automatic. Configuring any form parameters will cause the request to use a Content-Type header of application/x-www-form-urlencoded. If form data is present, any custom Content-Type header except multipart/form-data is ignored.

Success status By default, an HTTP response status code outside the 2XX range will result in a node error. Adding one or more status codes here overrides the default behavior, and causes the node to (only) consider the configured status codes as successful responses.

Body
A JSON request body can be explicitly configured by adding valid JSON data here. Dynamic values can be included with variable expression, but only in string values. For example:

{ "key": "${data:some-key||default-value}" }

Post-request code
Javascript code can be used directly in the HTTP client node to post-process response data and metadata. This capability is often used to:

1. Set state in the flow’s $data store, particularly useful for conditional execution in task flows.
2. Changing the shape of response data just before the HTTP client node produces it’s output data, by modifying the $out store.
3. Setting default data if the request fails
4. Overriding the error behavior of the HTTP client node, by modifying the value of the ok property in the default output data.
5. Change the default error message if the request fails

This code executes after the HTTP response completes, whether successfully or not. It has access to exactly the same stores as the JS Code node. The difference is that the $in store contains the node’s output data, instead of it’s input data.

Tips and Tricks

Simple solutions to common needs with HTTP requests.

Plaintext request bodies

The HTTP client node will send plain-text response bodies when:

1. It receives input data from another node
2. The input data is a JSON string (json:string)
3. It is configured with a Content-Type header, which does not match or contain application/json

Dynamic URLs

Assuming that a server variable exists with key api-url and value https://httpbin.org/get, use this in the URL configuration:

${vars:api-url}

Assuming the flow’s $data store contains a key host with value api.service.com, use this in the URL configuration:

https://${data:host}/some/path

Extract a header value

To place the Content-Type header in the flow’s $data store, use this as post-processing code:

if ($in.ok) {
  // request was OK
  $data.response_type = $in.response_headers?.["Content-Type"] || null;
} else {
  // request failed, use default
  $data.response_type = null;
}

Override the default user agent

The default User-Agent: sirveo request header is use in the absence of a custom User-Agent header. Configure a User-Agent header with your preferred value.

Dynamic header values

See Dynamic URLs. Variable expressions can be used as header values.

Dynamic headers

The HTTP client node does not directly support including or excluding headers according to dynamic criteria.

To achieve this behavior, use conditional execution in a task flow, with different HTTP client nodes.

Directly output response data

To make the HTTP client node directly produce response body data, use post-process code:

// set default value
$out = [];
// replace default if request was successful
if ($in.ok) $out = $in.body || $out;

Override error behavior

Typical problem: the success of an API request depends on the contents of it’s response data, not whether it responds with a 2XX status code.

Approach 1: throw an error in post-processing code

if ($in.ok) {
  // check something in response data (or headers)
  if (!($in.body && $in.body.itWorked === true)) {
    // cause in error
    throw new Error("request failed");
  }
}

Approach 2: modify the default ok property

if ($in.ok) {
  // check something in response data (or headers)
  if (!($in.body && $in.body.itWorked === true)) {
    // this will cause the node to emit output in it's error channel
    $out.ok = false;
  }
}

The second approach may also be used to prevent a node error, by setting $out.ok = true for failed requests.

Note that the HTTP client node only uses the value of $out.ok to determine success or error state when $out is an object. If $out contains something other than an object after post-processing code has run, the success/error state is determined by default behavior, which may be modified with custom success status code criteria.

Handling timeouts

To detect timeouts in post-processing code, or in subsequent JS Code nodes which receives the HTTP client node’s output:

if ($in.error && $in.error === "request timed out") {
  // do something
  $out.timeout = true;
  $data.timeout = true;
}

Measure latency

Place a JS Code node before the HTTP request runs.

// timestamp in milliseconds, before request
$data.ts_before = lib.NowMs();
$out = $data;

In the HTTP client node’s post-processing code:

// subtract for reasonably accurate round-trip time
$out.rtt_ms = lib.NowMs() - $data.ts_before;

// halve for approximation of one-way latency
$out.latency_ms = $out.round_trip_time / 2;

Output:

{
  "latency_ms": 709.5,
  "ok": true,
  "rtt_ms": 1419,
  "status_code": 200,
  "status_text": "200 OK",
  "uri": "https://httpbin.org/get"
}

Cache responses

Assuming that response data fits into the KV Store’s 5 KiB size limit, here’s the general pattern:

1. Decide on a prefix and key for persisting the data
2. Send the HTTP client node’s output into a KV store node, with an appropriate expiry (e.g. 5m)
3. Then add a KV Get node before the HTTP request, to lookup the key
4. Conditionally run the HTTP node, based on whether the value was found or not.

Task flows are ideal for implementing such conditional logic. To use a graph flow, a JS condition node will be needed.

Parallel HTTP requests

In a task flow:

1. add two or more HTTP client nodes in the same task level
2. Click on the task level’s “serial” option, to change it to parallel

If post-processing code modifies the flow’s $data store in parallel nodes, unpredictable behavior is practically guaranteed. The slowest parallel node will have the final say about the $data store modification.

Instead, route the output of parallel HTTP client nodes to nodes in a subsequent (and serial) task level.

Tip: Subflow nodes and Subflow loop nodes can be run in parallel, with task flows.

Constraints

Size constraint on response bodies

The HTTP client node will not process response bodies larger than 256 KiB, which is 256 * 1024 bytes. This constraint cannot be removed or configured.

It is firstly a stability guarantee when flows become complex, and especially when the server is executing multiple flows concurrently. The size constraint is secondly a security feature, which prevents memory exhaustion when HTTP responses may unexpectedly contain very large response bodies.

Larger data payloads is not a technical issue for Sirveo’s flow engine, but they present a technical challenge when testing flows in a browser context. This size constraint will likely be increased closer to version 1.0.0, and is currently very low priority on the development roadmap. If your use case would benefit from a higher response body size limit, let us know.

No file support

The HTTP client node is not designed for dealing with files. If or when support for file uploads and downloads via HTTP is implemented, it will be supported with dedicated HTTP nodes.

Limited plain-text response support

Non-JSON response data (like HTML, XML or other plain-text formats) will be made available as strings within a JSON encoding. It’s possible to perform string processing on such data with javascript code, but not always convenient.

No retry support

The HTTP client node does not support a retry capability. It’s easy to configure simple or complex retry behavior in a task flow, using a combination of conditional task levels, and the Wait node to create delays.