aws iot 핸즈온 워크샵 - 실습 6. 긴급 데이터를 kinesis streams으로 보내기...

For this lab were going to look at a combination of systems to capture different kinds of data streams. We'llstart with the usual AWS IoT system using rules to capture our data but during certain events, as determinedby our rules we'll activate a Direct-To-Kinesis stream to capture realtime information.

Why would you ever do this? AWS IoT has limits in place for the size of each message but also the number oftransactions per second. When streaming realtime telemetry information, if you find you need more than50TPS you would want to consider going to directly to Kinesis.

Let's setup a Kinesis Shard. Let's setup a single shard to capture our stream when it is activated.

Open the Kinesis dashboard and pick Kinesis Streams.Click Create Stream.Let's call this "iotshard", you'll enter just 1 shard.

Lab 9 - Switch to Kinesis during critical events

Step 1

We're going to setup some Lambda functions to handle logging for us so we can keep track of our "thing".

Open the Lambda dashboard.Click Create a lambda function.You can use the hello-world blue print.Let's call this function, "iotLambda2", accept the basics.You can also enter the code below to turn on some basic logging.

Step 2

Note: We are not creating these function in a VPC for this lab.

Next we need to create another Lambda function to capture our Kinesis stream data.

While in the Lambda dashboard, create a new function.Use the Kinesis-Process-Record blue print.Select the Kinesis Stream you created in step 1.Leave the other options as their defaults.We are going to call this iotLambda3.Make sure your role is a Kinesis Role.On the review screen, make sure you pick "Enable Now" so the event source is active.

Here is the code if you want to copy & paste:

console.log('Loading function');

exports.handler = function(event, context) { console.log('IoT Kinesis Event:', JSON.stringify(event, null, 2)); event.Records.forEach(function(record) { // Kinesis data is base64 encoded so decode here var payload = new Buffer(record.kinesis.data, 'base64').toString('ascii'); console.log('Decoded payload:', payload); }); context.succeed("Successfully processed " + event.Records.length + " records.");};

Lastly we need a Lambda function that can publish a change to our shadow state to turn on Kinesis for our"thing".

Create another new Lambda function, use the hello-world blue print.We will call this iotShadowUpdate.The code for this is shown below.

JavaScript

You will need to pick an execution role that has access to the IoT service.

Shadow Update Lambda code is as follows:

Open the AWS IoT Dashboard.Click create resource and pick a new rule.Let's call this iotCritical.We'll select * from our iotCritical topic.Pick the default action as Lambda.Pick iotLambda2 that we created earlier.

Let's now create the Kinesis rule.

Click create resource and pick a new rule again.Let's call this iotKinesis.We'll select * again from our iotCritical topic.We want to add a condition this time, temperature > 100.Pick the default action as Lambda.Pick the iotShadowUpdate we created earlier.

Step 4

Our script will now connect and send regular AWS IoT messages but one of these will be a high temperaturevalue that will trigger our state change, once we detect the change we'll stream to Kinesis.

Let's create a new script for this lab. We'll call this "iotCritical.js". This script is going to be using the full AWSSDK. If you don't have that installed you can run

npm install aws-sdk

We are also using a sleep library to run this script a little slower:

npm install sleep

First, create your bootstrap connection code as follows:

note: You'll need to add your Kinesis stream endpoint.

Step 5

Bash

Bash

if (x<100) { x++; normalMsg.temperature = 75; normalMsg.seq = x; // on our 10th message we'll pass the threshold just once and see if our shadow changes if (x==10) normalMsg.temperature = 110; setTimeout(function() { device.publish('iotCritical', JSON.stringify(normalMsg)); }, 500);

console.log("Our current stream state is " + streamState); console.log("Sent regular message " + x + " at temp: " + normalMsg.temperature //sleep.sleep(1); // seconds } else { console.log('Done!'); } }});

The above is pretty simple. We're first checking our streaming state and after that we're using the callbackfunctionailty to trigger a slower loop to keep sending messages to AWS IoT.

Once the critical temp is sent (see the line setting our high temp on the 10th message) we see the shadowchange to turn on our Kinesis stream.

Let's now add the stream() function which just blasts data to Kinesis:

device.on('message', function(topic, message) { console.log('message', topic, message.toString()); if (topic == "$aws/things/thingtest/shadow/update/delta") { var shadow = JSON.parse(message); streamState = shadow.state.streamState; // report back our state change msg.state.reported.streamState = streamState; device.publish('$aws/things/thingtest/shadow/update', JSON.stringify(msg)); if (streamState == 'on') { //startStream(); setTimeout(function() { stream(); }, 2500); } }

// Start our main loop if (topic == '$aws/things/thingtest/shadow/update/accepted') { // Let's loop slowly with regular messages normalMsg.temperature = 75; normalMsg.seq = x; device.publish('iotCritical', JSON.stringify(normalMsg)); console.log("Our current stream state is " + streamState); console.log("Sent regular message " + x + " at temp: " + normalMsg.temperature ) sleep.sleep(1); // seconds }

if (topic == 'iotCritical') { if (x<100) { x++; normalMsg.temperature = 75; normalMsg.seq = x; // on our 25th message we'll pass the threshold just once and see if our shadow changes if (x==10) normalMsg.temperature = 110; setTimeout(function() { device.publish('iotCritical', JSON.stringify(normalMsg)); }, 500);

console.log("Our current stream state is " + streamState); console.log("Sent regular message " + x + " at temp: " + normalMsg.temperature //sleep.sleep(1); // seconds } else { //process.exit(); console.log('Done!');

} }});

var stream = function() { console.log('Starting Kinesis Stream ...'); for(t=1;t<50;t++) { // Let's publish our telemtry data kinesisMsg.id = t; kinesisMsg.x = t; kinesisMsg.y = Math.random() * (5000 - 1) + 1; kinesisMsg.z = kinesisMsg.y/t; kinesis.putRecord({ StreamName : "iotshard", Data : JSON.stringify(kinesisMsg), PartitionKey : "id" }, function(err, data) { if (err) { console.log(err, err.stack); // an error occurred } else { console.log("Sent stream message to Kinesis: " + JSON.stringify } }); }}

device.on('connect', function() { console.log('Connected!'); setTimeout(function() { device.subscribe('$aws/things/thingtest/shadow/update/delta'); device.subscribe('$aws/things/thingtest/shadow/update/accepted'); device.subscribe('iotCritical'); device.publish('$aws/things/thingtest/shadow/update', JSON.stringify(msg)); console.log("Set our thing shadow."); console.log('Waiting ...'); }, 2500);});

You can run this now and you see the switch between AWS IoT and Kinesis.

node iotCritical.jsBash

aws iot 핸즈온 워크샵 - 실습 6. 긴급 데이터를 kinesis streams으로 보내기...

Travel