Sending and receiving an event by hand

This tutorial uses httpie as a convenient substitute for curl.

First, look up your billing id:

$ strm auth access-token
Billing-id demo8542234275
...

Next, create a stream:

$ strm create stream by-hand
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "by-hand"
  },
  "enabled": true,
  "limits": {
    "eventRate": "999999",
    "eventCount": "999999999"
  },
  "credentials": [
    {
      "clientId": "w0qu00hwl644b...",
      "clientSecret": "OygfdpwBqoekL..."
    }
  ]
}

Request an OAuth 2.0 id token:

idToken=$(http https://auth.strm.services/auth \
  billingId=demo8542234275\
  clientId=w0qu00hwl644bsnrpu2fm8sqtubf88\
  clientSecret='OygfdpwBqoekLMFJyQl(cUzeo)fcs3' | jq -r .idToken)

Some random data has been generated with this tool for the clickstream demo schema.

demo.json
{
  "strmMeta" : {
    "schemaId" : "",
    "nonce" : 0,
    "timestamp" : 0,
    "keyLink" : { "string" : "" },
    "billingId" : null,
    "consentLevels" : [0]
  },
  "producerSessionId" : "producerSessionId",
  "url" : "url",
  "eventType" : "click",
  "referrer" : "me",
  "userAgent" : "httpie",
  "conversion" : 1,
  "customer" : { "id" : "customer_id" },
  "abTests" : [ "test-a", "test-b"]
}
This is the json serialization format of Avro.Our client drivers use the much faster and more compact Avro binary format. NOTE: Later schema definitions have optional values for most of the items in strmMeta.

And use it to post some random data:

$ cat demo.json | http post https://in.strm.services/event \
        authorization:"Bearer $idToken" Strm-Schema-Id:clickstream

HTTP/1.1 400 Bad Request

regex clickstream/url 'url' doesnt match '^(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]'

This is as expected. Stream Machine gives an indication that a validation failed. This is an example of the mechanism that Stream Machine provides to indicate to the data producers that their data doesn’t conform to the rules of the Event Contract.

Modify the url field in demo.json to become any valid url (like https://streammachine.io, and try to send it again:

$ cat demo.json | http post https://in.strm.services/event \
    authorization:"Bearer $idToken" Strm-Schema-Id:clickstream
HTTP/1.1 204 No Content

204 is the http status code that Stream Machine returns when the event has been accepted and processed.

Curl instead of httpie

When using curl instead of httpie it is possible to observe the http/2 response, indicating the use of http/2 with its much higher throughput.

$ curl -v https://in.strm.services/event \
    -H "authorization: Bearer $idToken" \
    -H "Strm-Schema-Id:clickstream" --data-binary @demo.json

...
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
...
> POST /event HTTP/2
> Host: in.strm.services
...
> strm-schema-id:clickstream
> content-length: 434
>
...
< HTTP/2 204

Receiving events over websocket

Stream Machine provides a websocket event for development purposes only that allows one to see a JSON version of events as they arrive in the stream.

It’s nice to see some data when playing with the websocket. Use strm sim run-random stream-machine to generate random data on the stream-machine stream.

The Easy Way

The strm CLI has an option to read events from the websocket developer endpoint

$ strm egress --help
Read from egress

Usage:
  strm egress [flags]

Flags:
      --client-id string       client id to be used for sending data
      --client-secret string   client secret to be used for sending data
      --egress string          where to retrieve the events (default "wss://out.dev.strm.services/ws")

client-id and client-secret are optional, but required if you didn’t create the stream with the --save option.

$ strm egress stream-machine
{
  "strmMeta": {
    "schemaId": "clickstream",
    "nonce": 83482388,
    "timestamp": 1625820984604,
    "keyLink": "033441a9-5179-421f-ac7b-213e2a7c59aa",
    "billingId": "demo8542234275",
    "consentLevels": [ 0, 1 ]
  },
  "producerSessionId": "ARJUfomlNrnfOgucA7wdnIENGoWJqOa5GzdQFWGptko=",
  "url": "https://www.streammachine.io/rules",
  "eventType": "",
  "referrer": "",
  "userAgent": "",
  "conversion": 0,
  "customer": {
    "id": "ARJUfokrkqHD5/vO0wkTRRd2BkRJy3kz469vugpKlD2emCwNH5vBw9E="
  },
  "abTests": []
}

The Hard Way

Here the wscat tool is used to check it out. First generate an idToken just as above (or use the same one if it is still valid). Next, connect with wscat to the websocket endpoint.

wscat -H "authorization:Bearer $idToken" -c wss://out.strm.services/ws
Connected (press CTRL+C to quit)

< {
  "strmMeta": {
    "schemaId": "clickstream",
    "nonce": 870922078,
    "timestamp": 1616676469945,
    "keyLink": 986476622,
    "billingId": "hello0123456789",
    "consentLevels": [ 0 ]
  },
  "producerSessionId": "AVabqQMXJ4QyUcGhR3d2RIARjb4/J2EZ9uwmje9ZxqC2RMRkrjo=",
  "url": "https://streammachine.io",
  "eventType": "click",
  "referrer": "me",
  "userAgent": "httpie",
  "conversion": 1,
  "customer": { "id": "AVabqQNXK53Q331WKOnAFAX6Fu93mt0pz4br8JzKF1w=" },
  "abTests": [ "test-a", "test-b" ]
}

Send an event via the CLI and observe its reception over the websocket. Note the encryption of all fields labelled PII in the clickstream schema.

Decrypting data

First, create a decrypted stream:

$ strm create stream --derived-from by-hand --levels 0
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "by-hand-0"
  },
  "consentLevels": [ 0 ],
  "consentLevelType": "CUMULATIVE",
  "enabled": true,
  "linkedStream": "by-hand",
  "credentials": [
    {
      "clientId": "mn2dwjzrak2xw...",
      "clientSecret": "AEPXdSfZc4mLr..."
    }
  ]
}
$ strm egress by-hand-0
{
  "strmMeta": {
    "schemaId": "clickstream",
    "nonce": -1032207473,
    "timestamp": 1625821841410,
    "keyLink": "9a0808c7-88eb-4970-8e67-e05180780ce5",
    "billingId": "demo8542234275",
    "consentLevels": [ 0, 1, 2, 3 ]
  },
  "producerSessionId": "AXtSsOFVCH5rgemori2PCzBcxnn0PEl60Zd0Tpy/r/Y=",
  "url": "https://www.streammachine.io/rules",
  "eventType": "",
  "referrer": "",
  "userAgent": "",
  "conversion": 0,
  "customer": {
    "id": "customer-session-888"
  },
  "abTests": []
}
the customer/id field is decrypted (defined in the schema as consent level 0), but the producerSessionId is not! because that is of consent level 1. If the event had not contained 0 in its consent levels, we wouldn’t even have seen the event in this decrypted stream.

And finally, to clean up the resources:

$ strm delete stream by-hand --recursive
{
  "streamTree": {
    "stream": {
      "ref": {
        "billingId": "demo8542234275",
        "name": "by-hand"
      },
      "enabled": true,
      "limits": {
        "eventRate": "999999",
        "eventCount": "999999999"
      },
      "credentials": [
        { "clientId": "w0qu00hwl644bsnrpu2fm8sqtubf88" }
      ]
    },
    "keyStream": {
      "ref": {
        "billingId": "demo8542234275",
        "name": "by-hand"
      }
    },
    "derived": [
      {
        "ref": {
          "billingId": "demo8542234275",
          "name": "by-hand-0"
        },
        "consentLevels": [ 0 ],
        "consentLevelType": "CUMULATIVE",
        "enabled": true,
        "limits": {},
        "linkedStream": "by-hand",
        "credentials": [
          { "clientId": "mn2dwjzrak2xw6t9kbxtabkzwcyp8x" }
        ]
      }
    ]
  }
}