case study on aws javascript sdk and dynamodb

AWS JavaScript SDK

and DynamoDBCliff Chao-kuan Lu

<[email protected]>

A Case Study:

May 21st ‘14, AWS User Group Taiwan

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>

2

授權與格式說明以超連結 (hyperlink) 代替引用 (attribution)

引用外部內容均◦已取得授權，或◦包含原始連結，並在合理範圍內引用

本文件原創內容以 CC-BY 3.0 釋出

https://creativecommons.org/licenses/by/3.0/tw/


3

About cliffluFull-stack Web DeveloperAWS Solutions ArchitectNerd

about.me/clifflu

http://about.me/clifflu



4

About cliffluFull-stack Web DeveloperAWS Solutions ArchitectNerd

about.me/clifflu

Level Up !! Professional




5

Intro◦AWS JS SDK◦IAM◦DynamoDB

Case Study◦Headless Poller

大綱


6

Everything Changes


7

AWS SDK for JavaScript in the Browser

這名字好長

Part 1.1


8

很新AWS JS SDK / Browser

http://aws.typepad.com/aws/2013/10/developer-preview-aws-sdk-for-javascript.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AmazonWebServicesBlog+(Amazon+Web+Services+Blog)


9

官網起自 AWS SDK for Node.js2.0 更名並支援 Browser

AWS JS SDK / Browser

http://aws.amazon.com/developers/getting-started/browser/


10

率先支援 ◦DynamoDB◦S3◦SNS◦SQS

猛烈 rc 中

服務

https://aws.amazon.com/releasenotes/JavaScript?c=25&p=1&sm=dD


11

AutoScaling, CloudFormation, CloudFront, Cloudsearch, CloudTrail, CloudWatch, DataPipeline, DirectConnect, DynamoDB, EC2, ElastiCache, ElasticBeanstalk, ElasticTranscoder, ELB, EMR, Glacier, IAM, ImportExport, Kinesis, OpsWorks, RDS, Redshift, Route53, S3, SES, SimpleDB, SNS, SQS, StorageGateway, STS, Support, SWF

服務


12

支援

http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-intro.html




13

IE…


14

用例國防布


15

SDK for◦PHP, Python, Node.js◦Java, .NET, ◦Ruby

AWS SDK for Android & iOSDec. 8th, 2010

親友


16

多此一舉？


17

Browser 直通 AWS◦減低對 API Server 的需求

根本


18

機房、機器、網路、電力很難搞交給 Amazon Web Services 正好那 EC2 Instance 呢？

反思


19

EC2 各種麻煩◦Load Balancing◦Types◦Contracts◦AutoScaling Parameters Pattern

淵藪


20

太多 Gotcha 每層服務、各層之間都要考慮複雜度可能變 M x N

Scaling & HA


21

範例example.com

ELBRoute53

CF

S3StaticConte

nt

SharedEnv

Auto Scaling group

AMI

AZ 2Web

ServersS1

Secondary

S2 Secondary

Config

AZ 1Web Servers

S1 Primary

S2 Primary

Config

AZ 3Web Servers

Config + Arbitor

mongod


22

將麻煩留給 AWS◦第三方與服務端授權◦Scaling / HA

優點


23

促成符合 SOA Pattern 之架構優點Web Page

HTML

CSSJS

Authenticate &

Authorize

Services

1. Auth Request

3. Authorized Identity

2. Access Token


24

容易整合其他服務 IAM

STS: Security Token Service WIF: Web Identity Federation

DynamoDB, S3, … 自有服務， SOA 嘛 O.o/

優點


25

減低將 Access / Secret Token 打入源碼的可能性優點


26

API IAMServices

基石


27

RESTful, SOAP (deprecated)Dev Tools 是好朋友API


28

安全乃第一要務不然會變成礦工 ˇˇ然後


29

Identity and Access Management

AWS 權限樞紐

Part 1.2


30

Root UserGroup / IAM UserRoles

◦AWS SVC◦X-Account◦IdP Web SAML

IAM: Identity


31

Password MFA (Multi-factor authentication)

Access / Secret Key Pair X.509 certificate

3rd Party ◦SAML◦Web Identity Federation

IAM: Authentication


32

IAM: Policy


33

IAM: Policy Effect: Deny | Allow Action:

◦允許呼叫的 API Resource:

◦arn Principal

◦授權端限制 Condition

◦其他限制


34

for rule in rules:◦Explicit Deny -> Deny◦Explicit Allow -> Allow◦Default Deny

稽核


35

IAM: arn

格式◦冒號分隔◦首二節固定為 arn:aws◦Service◦Region◦Account◦Resource Identifier


36

給路人甲的◦Access / Secret Key Pair◦Management Console 登入權限

IAM : STS


37

第三方驗證，歡迎大家路過不用怕被關


38

Web Identity Federation

Facebook, Google 可也

http://docs.aws.amazon.com/STS/latest/UsingSTS/STSUseCases.html


39

只接受下列驗證機制◦表三家： Amazon, Facebook, Google◦SAML 說明列表 IAM Partners

限制

http://docs.aws.amazon.com/IAM/latest/UserGuide/IdP-solution-providers.html

http://aws.amazon.com/iam/partners/


40

Trust Relationships◦Identity Provider◦Client ID

Permissions

IAM Role for WIF


41

用戶可透過第三方驗證與 IAM:STS, WIF 授予調用 AWS API 之權限

至此


42

WIF PlaygroundLogin with amazonAWS Documentation

◦Using IAM◦Using STS◦SDK for JavaScript

參照

https://web-identity-federation-playground.s3.amazonaws.com/index.html

http://login.amazon.com/website


43

DynamoDBManaged NoSQL Service

Part 1.3


44

Managed NoSQL Service 取代 SimpleDB 三本柱

◦Scalable◦Available◦Fast

DynamoDB


45

Data ModelScalar

◦Number{“N”: “300”}

◦String{“S”: “300”}

◦Binary{“B”: “BASE64”}

Multi-valued◦Number Set

{“NS”: [“1”,”2”,”3.14”]}◦String Set

{“SS”: [“A”,”b”]}◦Binary Set

{“BS”: [“BASE64”]}


46

Document size: 64 KB (UTF-8) 5 LSIs / 5 GSIs per Table Min throughput: 1

Hash Key: 2 KB Range Key: 1KB

BatchGetItem: 1MB or 100 items BatchWriteItem: 1MB or 25 items

Limits


47

Data Storage:◦約是 S3 10x

Provisioned Throughputs◦Reads ： 4kb 循序可合併 Eventually Consistent 消耗減半

◦Writes : 1kb

Pricing


48

Hash Key [opt] Range Key

Primary Key

hash: “a”

hash: “b”

{“hash”: “a”, “range”: “123”, …}{“hash”: “a”, “range”: “223”, …}{“hash”: “a”, “range”: “321”, …}

{“hash”: “b”, “range”: “3”, …}{“hash”: “b”, “range”: “22”, …}{“hash”: “b”, “range”: “321”, …}


49

Primary Key

hash: “a”

hash: “b”

{“hash”: “a”, “range”: “123”, …}{“hash”: “a”, “range”: “223”, …}{“hash”: “a”, “range”: “321”, …}

{“hash”: “b”, “range”: “3”, …}{“hash”: “b”, “range”: “22”, …}{“hash”: “b”, “range”: “321”, …}

SortedSharded


50

Index Name

Hash KeyRange KeyProjection

Shared Throughputs

Local Secondary Index


51

Index Name / Hash / Range Key ◦比照 LSI◦不要求 uniqueness

有自己的 Throughputs

Eventual ConsistencyProjected Attributes

Global Secondary Index


52

針對特定 Index (PK, LSI or GSI) 查詢支援 Condition 支援 Filter 高效

Query


53

現適用於 scan 及 query 消耗 throughput 不變減低 DynamoDB <-> Caller 傳輸boto 僅 boto.dynamodb2 支援

Filter


54

…Scan


55

Upsert 需包含 Primary Key

Update: PATCHPut: POST (PUT)

支援 Conditional Operation

Put, Update


56

支援 Conditional Operation 用於刪除 Item

若要刪除 Attribute ，需使用 updateItem

Delete


57

“Expected” 可實現 MVCC pattern

配合 Document Atomicity 可模擬transactional behavior (2-Phase Commit)

Conditional Operation


58

updateItem◦AttributeUpdates Value Action:

PUT DELETE ADD

Atomic Increment


59

Fine-Grained Policies

Item

Attribute


60

Fine-Grained Policies

Item◦“dynamodb:LeadingKeys”: [“xxx”]

Attribute◦“dynamodb:Attributes”: [“xxx”,”yyy”]


61

While True: Item.save(expect=…)

1unit for Read / Write

More on Throughput


62

Stats from DynamoDB

More on Throughput


63

CloudWatch 顯示低階數據`Expect` 不消耗 Read Unit 允許 short burst

猜想


64

了解 DynamoDB 的◦設計目標◦調校 Index Throughput

◦操作◦計費

小結


65

Headless-Poller終於到正題了 !?

Part 2


66

Phil: clifflu 你要不要講五月小聚

Henry: 聽說 Cloudflare + wordpress 會爆炸clifflu: 好啊，就講這個吧

緣起


67

秒殺WP 看 location.href 重導頁面講不滿三十分鐘 ˇˇ 只好調出備用題目

然後


68

EC2 各種麻煩◦Load Balancing◦Types◦Contracts◦AutoScaling Parameters Pattern

回想


69

就來演示不靠 EC2 的小服務吧


70

使用服務◦IAM, S3 (DynamoDB), CF

Octopress / Jekyll !?Clone Ruby 好像很遜 >///<

寫個 Blog system ?


71

最新功能最冷門功能最好 Manual 沒有

必須霸氣外露！


72

前端◦Angular.js 潮◦OAuth 勁

後端◦從缺，帥

資料◦DynamoDB ，猛

線上投票


73

前端◦Angular.js 潮◦OAuth 勁

後端◦從缺，帥

資料◦DynamoDB ，猛

線上投票需要 https

CloudFront


74

申請 APP◦Amazon◦Facebook◦Google

將 access_token 透過 STS 轉換為 Access / Secret Key Pair

第三方授權


75

欄位： uid, q_id, o_id

Primary Key◦(uid, q_id) : unique 確保每人每題一票◦選 uid 為 hash key Authentication Cardinality

DynamoDB


76

updateItem (省略 callback)投票

Primary Key

Upsert

Return on Update


77

updateItem權限

Hash Key


78

正解： Worker ，但霸氣不足需求：

◦uid 不外漏◦affordable◦快

計票


79

建立 (q_id, o_id) 之 GSI禁止讀取 uid 欄位取出列數即為總票數循序讀取，節省 read capacity

計票： GSI


80

LSI / GSI 必定包含 Primary Key 透過 Query / Scan 取得 Item 時，必須允許讀取 primary key

麻煩，天大的麻煩


81

曙光


82

運用 “ select”: “COUNT” 只計算票數，不取 Item body

循序讀取

計票： GSI


83

Document 沒寫Boto 沒使用

◦dynamodb 實做了 query.count() ，透過取回 item 記數◦dynamodb2 未有類似功能

霸氣


84

GSI Read Capacity, 1.5 !!??穩定


85

權限


86

計票


87

終於


88

參數Throughpu

tsRead Write

Table 1 5GSI 19 5


89

參數Throughpu

tsRead Write

Table 1 5GSI 19 5

Total #Reads8kb data 消耗 1

投票消耗GSI ~ Table

~6.6 USD/mo


90

改用 Batched Query ?◦不支援 select COUNT

優化流程◦資料更新◦後台計票◦操作介面◦PR please

改進？


91

講完了今天沒有 bonus session


92

Q & A回家記得念 Manual


93

/clifflu/headless-poller

簡報下載： AWS Doc

◦IAM◦STS◦DynamoDB

API

Links

https://github.com/clifflu/headless-poller

https://github.com/clifflu/headless-poller

http://docs.aws.amazon.com/IAM/latest/UserGuide/IAM_Introduction.html

http://docs.aws.amazon.com/STS/latest/UsingSTS/Welcome.html

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/Welcome.html

https://github.com/

http://www.slideshare.net/clifflu666/case-study-on-aws-javascript-sdk-and-dynamodb

case study on aws javascript sdk and dynamodb

Software