Sending and receiving an event by hand

This tutorial uses httpie as a convenient substitute for curl.

First, look up your billing id:

$ strm auth show
...
Billing id = hello0123456789
...

Next, create a stream:

$ strm streams create by-hand
{
  "name": "by-hand",
  "tags": [],
  "credentials": {
    "clientId": "hc8davgkhy1y...",
    "clientSecret": "Jwi_HDHWD*7..."
  }
}

Request an OAuth 2.0 id token:

idToken=$(http https://auth.strm.services/auth \
    billingId=hello0123456789 clientId="hc8davgkhy1y..." clientSecret="Jwi_HDHWD*7..." | jq -r .idToken)

Some random data has been generated with this tool for the clickstream demo schema.

demo.json

{
  "strmMeta" : {
    "schemaId" : "",
    "nonce" : 0,
    "timestamp" : 0,
    "keyLink" : { "int" : 0 },
    "billingId" : null,
    "consentLevels" : [0]
  },
  "producerSessionId" : "producerSessionId",
  "url" : "url",
  "eventType" : "click",
  "referrer" : "me",
  "userAgent" : "httpie",
  "conversion" : 1,
  "customer" : { "id" : "customer_id" },
  "abTests" : [ "test-a", "test-b"]
}

this is the json serialization format of Avro.Our client drivers use the much faster and more compact Avro binary format. NOTE: later schema definitions have optional values for most of the items in strmMeta.

And use it to post some random data:

$ cat demo.json | http post https://in.strm.services/event \
        authorization:"Bearer $idToken" Strm-Schema-Id:clickstream

HTTP/1.1 400 Bad Request

regex clickstream/url 'url' doesnt match '^(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]'

This is as expected.Stream Machine gives an indication that a validation failed. This is an example of the mechanism that Stream Machine provides to indicate to the data producers that their data doesn’t conform to the rules of the Event Requirement Specification.

Modify the url field in the event to become any valid url, and try to send it again:

$ cat demo.json | http post https://in.strm.services/event \
    authorization:"Bearer $idToken" Strm-Schema-Id:clickstream
HTTP/1.1 204 No Content

204 is the http status code that Stream Machine returns when the event has been accepted and processed.

Curl instead of httpie

When using curl instead of httpie it is possible to observe the http/2 response, indicating the use of http/2 with its much higher throughput.

$ curl -v https://in.strm.services/event \
    -H "authorization: Bearer $idToken" \
    -H "Strm-Schema-Id:clickstream" --data-binary @demo.json

...
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
...
> POST /event HTTP/2
> Host: in.strm.services
...
> strm-schema-id:clickstream
> content-length: 434
>
...
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 204

Receiving events over websocket

Stream Machine provides a websocket event for development purposes only that allows one to see a JSON version of events as they arrive in the stream.Here the wscat tool is used to check it out. First generate an idToken just as above (or use the same one if it is still valid). Next, connect with wscat to the websocket endpoint.

$ wscat -H "authorization:Bearer $idToken" -c wss://out.strm.services/ws
Connected (press CTRL+C to quit)

< {
  "strmMeta": {
    "schemaId": "clickstream",
    "nonce": 870922078,
    "timestamp": 1616676469945,
    "keyLink": 986476622,
    "billingId": "hello0123456789",
    "consentLevels": [ 0 ]
  },
  "producerSessionId": "AVabqQMXJ4QyUcGhR3d2RIARjb4/J2EZ9uwmje9ZxqC2RMRkrjo=",
  "url": "https://streammachine.io",
  "eventType": "click",
  "referrer": "me",
  "userAgent": "httpie",
  "conversion": 1,
  "customer": { "id": "AVabqQNXK53Q331WKOnAFAX6Fu93mt0pz4br8JzKF1w=" },
  "abTests": [ "test-a", "test-b" ]
}

Send an event via and observe its reception over the websocket.Note the encryption of all fields labelled PII in the clickstream schema.

Decrypting data

First, create a decrypted stream:

$ strm outputs create -cl 0 -clt CUMULATIVE by-hand by-hand-0
{
  "linkedStream": "by-hand",
  "name": "by-hand-0",
  "tags": [],
  "credentials": {
    "clientId": "1rq9...",
    "clientSecret": "dc..."
  },
  "consentLevelType": "CUMULATIVE",
  "consentLevels": [ 0 ],
  "decrypterName": "default"
}

Create a new idToken as above.

this token gives access to this outputstream (by-hand-0), not to the input stream by-hand. Start the websocket, and send some data. Observe that the PII attributes are now decrypted.

idTokens are on a per stream basis.

$ wscat -H "authorization:Bearer $idToken" -c wss://out.strm.services/ws
Connected (press CTRL+C to quit)

< {
  "strmMeta": {
    "schemaId": "clickstream",
    "nonce": -1160853344,
    "timestamp": 1616676967864,
    "keyLink": 986476622,
    "billingId": "hello0123456789",
    "consentLevels": [ 0 ]
  },
  "producerSessionId": "AVabqQMXJ4QyUcGhR3d2RIARjb4/J2EZ9uwmje9ZxqC2RMRkrjo=",
  "url": "https://streammachine.io",
  "eventType": "click",
  "referrer": "me",
  "userAgent": "httpie",
  "conversion": 1,
  "customer": { "id": "customer_id" },
  "abTests": [ "test-a", "test-b" ]
}

the customer/id field is decrypted (defined in the schema as consent level 0), but the producerSessionId is not! because that is of consent level 1. If the event had not contained 0 in its consent levels, we wouldn’t even have seen the event in this decrypted stream.

And finally, to clean up the resources:

$ strm outputs delete by-hand by-hand-0
$ strm streams delete by-hand