Exporting encryption keys

The output streams feature manages the decryption of data for you. The keys exist only within the Stream Machine keys database, for a duration a little longer than the keys rotation period [1]. The output streams have a default retention of 7 days, so if you don’t have the keys, and don’t consume or export the output streams within 7 days, you lose the capability to decrypt the personal data attributes.

Assuming your company decides that it wants to have the actual encryption keys [2], you need to configure Stream Machine to provide you with the keys. A pre-requisite is that your account is enabled for this capability. If not the features below will be forbidden to you.

Exporting keys is only permitted, if your account allows this.

The Encryption Keys

We use Google Tink as an abstraction library for standard AES-256 encryption with a synthetic initialization vector.

Creating an exporter

Currently, we only provide batch exporters for the encryption keys, that work very similar to the events batch exporters. So you need the same mechanism with authenticated and authorized IAM users.

$ strm create batch-exporter --help
Create batch exporter

Usage:
  strm create batch-exporter [stream-name] [flags]

Flags:
      --export-keys          Do we want to export the keys stream
  -h, --help                 help for batch-exporter
      --interval int         Interval in seconds between batches (default 60)
      --name string          optional batch exporter name
      --path-prefix string   path prefix on bucket
      --sink string          name of the sink. Optional if you have only one defined sink.

We’re looking for the --export-keys option. Provided key exporting is enabled for your account, you can do the following:

$ strm create batch-exporter stream-machine \
  --export-keys \
  --interval 30 \
  --path-prefix stream-machine-keys | jq
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "s3-stream-machine-keys"
  },
  "keyStreamRef": {
    "billingId": "demo8542234275",
    "name": "stream-machine"
  },
  "interval": "30s",
  "sinkName": "s3",
  "pathPrefix": "stream-machine-keys"
}
If you have more than 1 sink defined, you must give the name of that sink.
The current implementation (released on 04 May 2021) of this key export mechanism does not export keys that were created more than 7 days earlier.

Exported keys in S3

$ aws s3 ls stream-machine-export-demo/perf-test-keys/
2021-05-04 15:41:37          0 .strm_test...95-dfec21be8251.jsonl (1)
2021-05-04 16:13:01     166008 2021-05-04T14:13:00-keys-e1...-7-8-9.jsonl (2)
2021-05-04 16:13:31     701824 2021-05-04T14:13:30-keys-e1...-7-8-9.jsonl

$ aws s3 cp \
  s3://stream-machine-export-demo/perf-test-keys/2021-05-04T14:13:00-keys-e1...-7-8-9.jsonl \
  - | head -1

{ "keyLink": "44861053-6a95-4ec6-8b33-96fd1f748402", (3)
  "tinkKey": {"primaryKeyId":84683988,"key":[
    {"keyData":{"typeUrl":"type.googleapis.com/google.crypto.tink.AesSivKey",
    "keyMaterialType":"SYMMETRIC",
      "value":"EkDzauIHozdnF.....WkpB8Xu"}, (4)
      "outputPrefixType":"TINK","keyId":84683988,"status":"ENABLED"}]}
}
1 This is a test file created by Stream Machine to verify that we can actually write in this bucket. Because it starts with a . it is ignored by most tools.
2 Because the interval is 30 seconds, we’ll have a file every 30 seconds. Each file contains json lines with one key per line. The line contains both a keyLink attribute, with the key link of the events, and a tinkKey attribute that contains the serialized Tink key. The format is described in this protobuf file.
3 the key link that exists on all Stream Machine events.
4 the actual AES-256 encryption key.

1. default 24 hours
2. with the associated security and personal data hassles!