aws black belt techシリーズ amazon sns / amazon sqs

Amazon Simple Queue Service(SQS)Amazon Simple Notication Service(SNS)

AWS Black Belt Tech Webinar 2015

201534

! NameEiji Shinohara ( )

! RoleSolutions Architect for StartupsAmazon CloudSearch Subject Matter Expert

Who am I?

Amazon Simple Queue Service (SQS) Amazon Simple Notication Service (SNS) SQSSNS

Amazon Simple Queue Service (SQS)

https://aws.amazon.com/blogs/aws/amazon_simple_q/

2006713

Message Queue

MQ http://ja.wikipedia.org/wiki/

Amazon SQSPullMQ SQS

polling

Amazon SQSAWS : / : / : : +

Massive Message Processing with Amazon SQS and Amazon DynamoDB (ARC301) | AWS re:Invent 2013

SQS: Loosely Coupled

: :

150

Amazon SQS

SQS: Loosely Coupled Amazon SQSHappy

OK

JavaSQSAWS SDK

2

JavaPHP

PHP

Amazon SQS

Management Console

Amazon SQS

Management Console :

Amazon SQS

Management Console

Amazon SQS

Management Console :

Amazon SQS

Management Console - : Message Body

Amazon SQS

Management Console -

Amazon SQS Queue URL

URL https://.queue.amazonaws.com/ID/

Message ID ID

Receipt Handle Visibility

Visibility

Amazon SQS

AWS CLI

$ aws sqs create-queue --queue-name blackbelt2015{ QueueUrl: https://.queue.amazonaws.com/ID/blackbelt2015"}

Amazon SQS

AWS CLI :

$ aws sqs list-queues{ "QueueUrls": [ https://.queue.amazonaws.com/ID/blackbelt2015" ]}

Amazon SQS

AWS CLI

$ aws sqs send-message --queue-url https://.queue.amazonaws.com/ID/blackbelt2015 --message-body 'Hello World!'{ "MD5OfMessageBody": "ed076287532e86365e841e92bfc50d8c", "MessageId": "2db5a1de-8bc5-4976-9b0c-68749bee1a"}

Amazon SQS

AWS CLI

$ aws sqs receive-message --queue-url https://.queue.amazonaws.com/ID/blackbelt2015{ "Messages": [ { "Body": "Hello World!", ReceiptHandle: AQEBv3p27X8o44Nt4J0 +8lujWoy6Ks=", "MD5OfBody": "ed076287532e86365e841e92bfc50d8c", "MessageId": "2db5a1de-8bc5-4976-9b0c-68749bee1a" } ]}

Amazon SQS

AWS CLI

aws sqs delete-message --queue-url https://.queue.amazonaws.com/ID/blackbelt2015 --receipt-handle "AQEBv3p27X8o44Nt4J0 +8lujWoy6Ks="

Amazon SQS AWS SDK

AmazonSQS sqs = new AmazonSQSClient();CreateQueueRequest createQueueRequest = new CreateQueueRequest("MyQueue");String myQueueUrl = sqs.createQueue(createQueueRequest).getQueueUrl();

sqs.sendMessage(new SendMessageRequest(myQueueUrl, Hello World! "));

Amazon SQS AWS SDK

ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);List messages = sqs.receiveMessage(receiveMessageRequest).getMessages();for (Message message : messages) {

// }

String messageRecieptHandle = messages.get(0).getReceiptHandle();sqs.deleteMessage(new DeleteMessageRequest(myQueueUrl, messageRecieptHandle));

Amazon SQS AWS SDK

AWS SDK20153

Java http://aws.amazon.com/jp/sdk-for-java/ Ruby http://aws.amazon.com/jp/sdk-for-ruby/ Python http://aws.amazon.com/jp/sdk-for-python/ .NET http://aws.amazon.com/jp/sdk-for-net/ PHP http://aws.amazon.com/jp/sdk-for-php/ JavaScript http://aws.amazon.com/jp/sdk-for-browser/ Node.js http://aws.amazon.com/jp/sdk-for-node-js/ Go(Coming Soon!) https://github.com/awslabs/aws-sdk-go

sqs.deleteQueue(new DeleteQueueRequest(myQueueUrl));

Amazon SQS

(SQS) 100 SQS1000.476 USD

1GB/0 USD 350TB /0.120 USD GB 350TB SQSEC2

Amazon SQS Visibility Timeout + EC2 Spot Instance API10/ Long PollReceive Message

http://www.slideshare.net/AmazonWebServices/advanced-topics-session-4-architecting-forhighavailability

Amazon SQS

Amazon SQS Visibility Timeout

1

2

3

Amazon SQS


1

1

2

3

Amazon SQS


30(12)23(in Flight)

1

2

3

Amazon SQS


1

1

2

3

Amazon SQS Amazon EC2 Spot Instances

Amazon EC2

201538

>= < 1

Amazon SQS Visibility Timeout + EC2 Spot Instances

Spot Instance1Visibility Timeout

:

Amazon SQS

1

2

3

Amazon SQS /1MAX10

SQS

Amazon SQS

1

2

3

1

2

3

x 10

x 10

x 10

x 10

x 10

x 10

Amazon SQS /1MAX10

: AWS CLI send-message-batch $ aws sqs send-message-batch--queue-url https://.queue.amazonaws.com/ID/blackbelt2015 --entries '[{ "Id": "01", "MessageBody": "test message 01" },{ "Id": "02", "MessageBody": "test message 02" },]{ "Successful": [ { "MD5OfMessageBody": "fa27aa462b939f0c8cf67189198f1b2d", "Id": "01", "MessageId": "041d1604-1c80-4440-90bf-598db00cf752" }, ...

Amazon SQS /1MAX10

: AWS CLI receive-message --max-number-of-messages(110)

$ aws sqs receive-message--queue-url https://.queue.amazonaws.com/ID/blackbelt2015 --max-number-of-messages 10{ "Messages": [ { "Body": "test message 01", "ReceiptHandle": "AQEB 7g/jA=", "MD5OfBody": "fa27aa462b939f0c8cf67189198f1b2d", "MessageId": "041d1604-1c80-4440-90bf-598db00cf752" },

Amazon SQS

AWS SDK for Java AmazonSQSAsyncClient 10

maxBatchOpenMs: maxBatchSize: 1

Amazon SQS Long Polling

SQSReceive Message

(020)

Amazon SQS Long Polling Short Polling

Long Polling (CPU, ) Polling

Receive MessageShort Polling Long Polling

Polling

Polling

Polling Polling

Webinar SQS Message Attributes AWS CloudTrail Delete All Messages in SQS Queue SQS Client Library for Java Messaging Service (JMS)

Webinar Message Attributes 201456

Body10 NameType(String/Number/Binary)Value

Type(: oat, int)

Webinar AWS CloudTrail 2014716

APIS3 CloudTrail20147

Webinar

Delete All Messages in SQS Queue 2014128 PurgeQueue

Webinar

SQS Client Library for Java Messaging Service (JMS) 20141229 JMS: JavaAPI SQSJMS

JMS

Amazon SQS Amazon SQS

(1) SQS/

Amazon SQS

46014

In Flight(Visibility Timeout) 120,000In Flight OverLimit

256KB 2013664KB256KB

S3

Delay Queue Message Timers

Dead Letter Queue SQS

Elastic Beanstalk: Worker Tier

AWS Identity and Access Management (IAM)

CloudWatch

Delay Queue

0900

Message Timers

0900 Message TimersDelay Queue

Send Message Request

Delay Queue 30

Message Timers 60

Dead Letter Queue

(11000)(Dead Letter Queue)

0900

Dead Letter Queue

AWS Elastic Beanstalk: Worker Tier

SQS Auto Scaling

Sqsd(deamon)

Elastic Beanstalk Application

http://localhost:80/xxx

EC2 Instance

Auto Scaling group

CloudWatch

Auto Scaling

Job Observer

http://aws.clouddesignpattern.org/index.php/CDP:Job_Observer AWS

http://aws.typepad.com/aws_japan/2015/01/auto-scaling-with-sqs.html

AWS Identity and Access Management (IAM)

/ { "Version":"2012-11-05", "Id":"cd3ad3d9-2776-4ef1-a904-4c229d1642ee", "Statement" : [ { "Sid":"1", "Eect":"Allow", "Principal" : { "aws": "111122223333" }, "Action":["sqs:SendMessage","sqs:ReceiveMessage"], "Resource": "arn:aws:sqs:us-east-1:444455556666:queue2", "Condition" : { "IpAddress" : { "aws:SourceIp":"10.52.176.0/24" }, "DateLessThan" : { "aws:CurrentTime":"2009-06-30T12:00Z

CloudWatch

NumberOfMessageSent

SentMessageSize

NumberOfMessageReceived ReceiveMessage

NumberOfEmptyReceives ReceiveMessage

NumberOfMessagesDeleted

ApproximateNumberOfMessageDelayed DelayDelay QueueDelay

ApproximateNumberOfMessageVisible

ApproximateNumberOfMessageNotVisible visibility winodowend

Amazon Simple Notication Service (SNS)

Amazon SNS

Publish

1.Topic

2.

Amazon SNS

Publisher Topic

HTTP(S)

EMAIL

SQS

Mobile Push

Subscriber

Amazon Simple Notication Service (SNS) Mobile Push(2015311)AWS Black Belt

Tech Webinar

Amazon SNS Mobile Push

Apple Devices

Google Devices

Amazon Kindle Fire Devices

Amazon SNS Amazon SNS :

: 100 Email/Email-JSON: 1,000 HTTP/HTTPS: 100,000 Simple Queue ServiceSQS:

(64 KB 1) : 1000.5 USD Email/Email-JSON: 100,0002 USD HTTP/HTTPS: 1000.6 USD Simple Queue ServiceSQS:

SMS

Amazon SNS Management Console Mobile Services

Amazon SNS1. Topic

2. TopicSubscribe3. TopicPublish

Amazon SNS1. Topic

TopicARN(Amazon Resource Name)

Amazon SNS2. TopicSubscribe

EndpointConrm

Amazon SNS3. TopicPublish

Publish

(Email)

Amazon Simple Notication Service (SNS)

AWS

Amazon CloudWatch Billing Alert

Amazon SES Bounce/Complaint

Amazon S3

Amazon Elastic Transcoder /

Amazon SNS Amazon CloudWatchBilling Alert

Amazon SNS

Amazon SESBounce/Complaint

Amazon SNS

Amazon S3

Amazon SNS Amazon Elastic Transcoder/

HTTPS enhancements Server Name Indication (SNI) Basic Digest

Message Attributes Time To Live (TTL)

Webinar

HTTPS enhancements 2014428 Server Name Indication (SNI)

SNISSL/TLSSSLIP

http://ja.wikipedia.org/wiki/Server_Name_Indication Basic Digest

HTTP POST HTTPS URL http://www.rfc-editor.org/info/rfc2617

Webinar

Message Attributes 2014612 String, Number, Binary Mobile PushReserved Message Attributes

Baidu MPNS WNS

Webinar

TTL 2014710 Time To Live

Webinar

Cloud Design Pattern: Fanout http://aws.clouddesignpattern.org/index.php/CDP:Fanout

SQSSNS

S3 RDS DynamoDB

SQSSNS

WebSNS

SNSSQS

Amazon S3

Amazon RDS

DynamoDB

SQS

SNSSQS (64 KB 1)

Simple Queue ServiceSQS:

SQSSNS

WebSNS

SNSSQS

Amazon S3

Amazon RDS

DynamoDB

SQS

Amazon SQS

Amazon SNS

AWS

Amazon SQS Document

http://aws.amazon.com/jp/documentation/sqs/ Amazon SQS FAQ

http://aws.amazon.com/jp/sqs/faqs/ Amazon SNS Document

http://aws.amazon.com/jp/documentation/sns/ Amazon SNS FAQ

http://aws.amazon.com/jp/sns/faqs/ AWS

http://aws.clouddesignpattern.org/index.php Massive Message Processing with Amazon SQS and Amazon DynamoDB

(ARC301) | AWS re:Invent 2013 https://www.youtube.com/watch?v=RJCpjRTvo_w

Webinar

AWS http://aws.amazon.com/jp/aws-jp-introduction/

Twitter/FacebookAWS

@awscloud_jp

http://on.fb.me/1vR8yWm

aws black belt techシリーズ amazon sns / amazon sqs

Technology