Streamsets renew JWT token to call api

Many JWT tokens expire hourly and need to be renewed to pass in an api call. Streamsets auto renewal of JWT tokens may not work so here is another way to renew JWT tokens.


PIPELINE-1: A continuously running separate pipeline will periodically renew the JWT token and store in a text file

PIPELINE-2: The second pipeline then will read the new token from the file and call the api during every polling interval.

PIPELINE-1 definition:

We need to create a directory in the folder called resources:

$ mkdir /opt/streamsets-datacollector/resources

$ chown sdc:sdc /opt/streamsets-datacollector/resources

This pipeline consists of two stages: Dev Data Generator -> Shell executor

In the Dev Data Generator we set Field Name=dummyfield; Delay Between batches=3000000 (for 50min renewal of JWT token); Batch size=1; Number of threads=1

In the Shell executor we put a curl command as below:

echo curl -X POST '' | python -c “import sys, json; print json.load(sys.stdin)[‘jwt’]” > /opt/streamsets-datacollector/resources/access_token.txt

PIPELINE-2 definition:

In this pipeline we will pass the access token in the file /opt/streamsets-datacollector/resources/access_token.txt in the Authorization Bearer header. On every polling of the api the new access_token file is read and new token is used.

In the HTTP Client we use the Headers:

Authorization Bearer ${runtime:loadResource(‘access_token.txt’,false)}

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.