Exporting encryption keys

The output streams feature manages the decryption of data for you. The keys exist only within the Stream Machine keys database, for a duration a little longer than the keys rotation period [1]. The output streams have a default retention of 7 days, so if you don’t have the keys, and don’t consume or export the output streams within 7 days, you lose the capability to decrypt the personal data attributes.

Assuming your company decides that it wants to have the actual encryption keys [2], you need to configure Stream Machine to provide you with the keys. A pre-requisite is that your account is enabled for this capability. If not the features below will be forbidden to you.

Exporting keys is only permitted, if your account allows this.

The keys

We use Google Tink as an abstraction library for standard AES-256 encryption with a synthetic initialization vector.

Creating an exporter

Currently, we only provide batch exporters for the encryption keys, that work very similar to the events batch exporters. So you need the same mechanism with authenticated and authorized IAM users.

$ strm exporters create --help

Usage: strm exporters create [OPTIONS] stream-name

  Create a new exporter

Options:
  ...
  -k, --export-keys             Export the encryption keys

Arguments:
  stream-name  Name of the stream that is being exported

We’re looking for the --export-keys option. Provided key exporting is enabled for your billing, you can do the following:

strm exporters create perf-test --exporter-name perf-test-keys \
  --sink-name s3-export --sink-type s3 --interval 30 \
  --path-prefix perf-test-keys --export-keys
The current implementation (released on 04 May 2021) of this key export mechanism does not export the already existing keys. Only new keys will be exported. Because keys don’t last more than 24 hours, after 24 hours, you will have all the keys.

Exported keys in S3

$ aws s3 ls stream-machine-export-demo/perf-test-keys/
2021-05-04 15:41:37          0 .strm_test...95-dfec21be8251.jsonl (1)
2021-05-04 16:13:01     166008 2021-05-04T14:13:00-keys-e1...-7-8-9.jsonl (2)
2021-05-04 16:13:31     701824 2021-05-04T14:13:30-keys-e1...-7-8-9.jsonl

$ aws s3 cp \
  s3://stream-machine-export-demo/perf-test-keys/2021-05-04T14:13:00-keys-e1...-7-8-9.jsonl \
  - | head -1

{ "keyLink": "44861053-6a95-4ec6-8b33-96fd1f748402", (3)
  "tinkKey": {"primaryKeyId":84683988,"key":[
    {"keyData":{"typeUrl":"type.googleapis.com/google.crypto.tink.AesSivKey",
    "keyMaterialType":"SYMMETRIC",
      "value":"EkDzauIHozdnF.....WkpB8Xu"}, (4)
      "outputPrefixType":"TINK","keyId":84683988,"status":"ENABLED"}]}
}
1 This is a test file created by Stream Machine to verify that we can actually write in this bucket. Because it starts with a . it is ignored by most tools.
2 Because the interval is 30 seconds, we’ll have a file every 30 seconds. Each file contains json lines with one key per line. The line contains both a keyLink attribute, with the key link of the events, and a tinkKey attribute that contains the serialized Tink key. The format is described in this protobuf file.
3 the key link that exists on all Stream Machine events.
4 the actual AES-256 encryption key.

1. default 24 hours
2. with the associated security and personal data hassles!