Creating Streams with the Command Line Interface

The Stream Machine CLI is a command line interface for interacting with it. You can find its source code and binary at github.com/streammachineio/cli

The Stream Machine portal currently provides only limited functionality. You will need to use the CLI.

The CLI is nothing more than a thin interface to the Stream Machine gRPC interface.

Logging in

First create an account at streammachine.io and use the email and password to log in with the CLI.

$ strm auth login demo@streammachine.io
Enter password:
billingId demo8542234275
saved login to ~/.config/stream-machine/strm-creds-api.streammachine.io.json

This creates a file that contains a valid OAuth2.0 id and refresh token, in the application configuration directory [1]. The billingId is a shortened identifier that is derived from your email address.

You can use strm auth access-token to find the billingId that has been assigned to you. You need this with the client drivers.

$ strm auth access-token
eyJ...iLCJ0eXAiOiJKV1QifQ.eyJp....abcN_F
Expires at 2021-07-08 15:55:19 +0200 CEST
billingId demo8542234275

Listing streams

Streams can be listed and will be shown as JSON formatted.

When listing streams, the clientSecret is not shown. The secret is only shown once, which is upon creation of the stream.
$ strm list streams
{
  "streams": [
    {
      "stream": {
        "ref": { "billingId": "demo8542234275", "name": "stream-machine" },
        "enabled": true,
        "limits": {
          "eventRate": "99", "eventCount": "999999"
        },
        "credentials": [ {
            "clientId": "ylbt4v9o6dvvc..."
          } ]
      }
    },
...

Creating a stream

A stream can be created as follows:

$ strm create stream stream-machine
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "stream-machine"
  },
  "enabled": true,
  "limits": {
    "eventRate": "99",
    "eventCount": "999999"
  },
  "credentials": [
    {
      "clientId": "ylbt4v9o6dvvc...",
      "clientSecret": "M0fBiQnKNXn*U..."
    }
  ]
}

The billingId, clientId and clientSecret triplet is what identifies your stream when you send data to Stream Machine. Stream Machine uses the OAuth 2.0 client credentials flow to generate a bearer token that needs to be provided with each HTTP request. Our drivers have tooling to create and refresh these tokens, but nothing precludes you from creating the headers by hand (see the chapter about sending via curl to do this manually).

So with this information, you have enough information to start sending data to in.strm.services/event. With the same credentials you can connect to the egress endpoint with a websocket client to receive the events as you send them.

Creating decrypted streams

If you want to have Stream Machine decrypt data with certain consent levels, you need to create an output stream.

$ strm create stream --help
Create a stream

Usage:
  strm create stream [flags]

Flags:
      --consent-type string   CUMULATIVE or GRANULAR (default "CUMULATIVE")
  -D, --derived-from string   name of stream that this stream is derived from
      --description string    description
  -h, --help                  help for stream
  -L, --levels int32Slice     comma separated list of integers for derived streams (default [])
      --save                  save the result in the config directory
      --tags strings          tags

So let’s create one, with two consent levels, and a granular consent level type interpretation.

$ strm create stream --derived-from stream-machine --levels 0,1 --consent-type GRANULAR
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "stream-machine-0-1"
  },
  "consentLevels": [ 0, 1 ],
  "consentLevelType": "GRANULAR",
  "enabled": true,
  "linkedStream": "stream-machine",
  "credentials": [
    {
      "clientId": "11jvxvpy1e6jl...",
      "clientSecret": "tJkhj8lT9ybAA..."
    }
  ]
}

The derived stream is provided with a default name stream-machine-0-1 because we did not provide an explicit name. If we had added a name after the strm create stream we would have created a stream with that name.

So the derived stream named stream-machine-0-1 captures data from encrypted stream stream-machine (for the current billingId). It will drop all events that don’t at least have consent levels 0 and 1 in the event. Another way of defining decrypted streams is with consent level type cumulative. This means that the decrypted stream has just one consent level, and it will accept all events that have at least that consent level. It will decrypt PII fields up to and including the decrypted stream consent level. Cumulative is actually the default for creating derived streams.

$ strm delete stream stream-machine-0-1
{
  "streamTree": {
    "stream": {
      "ref": {
        "billingId": "demo8542234275",
        "name": "stream-machine-0-1"
      },
      "consentLevels": [ 0, 1 ],
      "consentLevelType": "GRANULAR",
      "enabled": true,
      "limits": {},
      "linkedStream": "stream-machine",
      "credentials": [
        {
          "clientId": "11jvxvpy1e6jl..."
        }
      ]
    }
  }
}

Note the streamTree field might also contain all the items derived from a source stream, like exporters.

$ strm create stream --derived-from stream-machine --levels 1
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "stream-machine-1"
  },
  "consentLevels": [
    1
  ],
  "consentLevelType": "CUMULATIVE",
  "enabled": true,
  "linkedStream": "stream-machine",
  "credentials": [
    {
      "clientId": "vnfku72pl3bgx...",
      "clientSecret": "UMkNFnKt8ly#F..."
    }
  ]
}

This stream named stream-machine-1 will contain the identical subset of events as stream-machine-0-1


1. ~/.config/stream-machine