Exporting via a Kafka Consumer

Data can be consumed from your streams [1] in JSON format via an internet accessible authorized access to our Kafka export cluster.

Create an exporter

First you need to create an exporter. This creates a kafka-exporter and one associated kafka-user. The user credentials can be used to access the topic.

$ strm create kafka-exporter stream-machine
{
  "ref": {
    "billingId": "demo8542234275",
    "name": "shared-export-stream-machine"
  },
  "streamRef": {
    "billingId": "demo8542234275",
    "name": "stream-machine"
  },
  "target": {
    "clusterRef": {
      "billingId": "internal",
      "name": "shared-export"
    },
    "topic": "export-f08..."
  },
  "users": [
    {
      "ref": {
        "billingId": "demo8542234275",
        "name": "service-account-export-53f..."
      },
      "kafkaExporterName": "kafka-exporter-f08cb...",
      "topic": "export-f08cb8...",
      "clientId": "export-53f190...",
      "clientSecret": "f327b959-45c7...",
      "clusterRef": {
        "billingId": "internal",
        "name": "shared-export"
      }
    }
  ]
}

You can see the created user credentials. The kafka exporter is named shared-export-stream-machine. shared-export is the name of the Kafka cluster owned by Stream Machine that we export to; your Kafka consumer will consume from this cluster.

In the future, we will have the capability to export to a Kafka Cluster owned by the customer.

Kafka users are entities that can consume from a Kafka Cluster. They have credentials, and a Kafka topic that they can consume from.

$ strm list kafka-users  shared-export-stream-machine
{
  "kafkaUsers": [
    {
      "ref": {
        "billingId": "demo8542234275",
        "name": "service-account-export-4c9..."
      },
      "kafkaExporterName": "shared-export-stream-machine",
      "topic": "export-131d0...",
      "clientId": "export-4c9e2...",
      "clientSecret": "b779d0...",
      "clusterRef": {
        "billingId": "internal",
        "name": "shared-export"
      }
    }
  ]
}

Consuming

Clone the Python Kafka Consumer and go into the directory and create a file config.ini. Fill out the values from the JSON above:

[kafka]
bootstrap_servers = export-bootstrap.kafka.strm.services:9092
topic = export-7c163d...
client_id = export-02e5c...
secret = 748e6b...
token_uri = https://auth.strm.services/token
group = demo

Next, install the Python dependencies:

$ python -m venv
$ . venv/bin/activate
$ pip install -r requirements.txt

And run the consumer:

$ python consumer.py

{'strmMeta':
    {'schemaId': 'nps_unified_v1',
    'nonce': -956233225,
    'timestamp': 1616749246981,
    'keyLink': -1225595339,
    'billingId': 'hello0123456789',
    'consentLevels': [7, 3]},
    'brand_source': '', 'platform': '', 'os': '',
    'version': '',
    'device_id': 'AXo+XdyK3A+QbVUe2+KoxqBumttxNZCbrNvhHHFK',
    'customer_id': 'AXo+XdxTFu26cu47dnYUvEKjxUqdsNk1kKrFOA/CjRI=',
    ...

Deleting the Kafka-Exporter

If you just try to delete the Kafka Exporter, you’ll get an error.

$ strm delete kafka-exporter shared-export-stream-machine
Error: rpc error: code = FailedPrecondition desc = Cannot delete billing_id:
"..."
name: "shared-export-stream-machine"
.name because it has users attached to it

So let’s do that:

$ strm delete kafka-user service-account-export-4c9e28d2-a793-44d9-9afe-2921f9700758
{"billingId":"demo8542234275","name":"service-account-export-4c9e28d2-a793-44d9-9afe-2921f9700758"}

We could also just have done a recursive delete with strm delete kafka-exporter shared-export-stream-machine --recursive


1. encrypted and decrypted