[awsマイスターシリーズ] amazon dynamodb

78
© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. AWS マイスターシリーズ Amazon DynamoDB 2013.10.02 アマゾン データサービス ジャパン株式会社 ソリューションアーキテクト 今井 雄太 re:Generate

Upload: amazon-web-services-japan

Post on 25-May-2015

7.826 views

Category:

Technology


0 download

TRANSCRIPT

  • 1. re:G eneAWS rateAmazon DynamoDB 2013.10.02 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. ! NoSQLRDB ! DynamoDB ! DynamoDB ! ! When to use ! 2 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 3. NoSQLRDB3 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 4. NoSQL! ! ! !MongoDB Riak HBase Cassandra! Neo4j ! CounchDB ! DynamoDBhttp://ja.wikipedia.org/wiki/NoSQL 4 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 5. NoSQL vs RDB! NoSQL BASE ! RDB ACID Atomicity Consistency Isolation Durability 5 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 6. BASE?6 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 7. Basically Availabl, Soft state, Eventually consistent ! RDBACID ! ! NoSQLBASE ACIDRDB 7 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 8. NoSQL 8 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 9. NoSQLMySQL1,000QPS NoSQL 9 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 10. RDB NoSQL 10 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 11. DynamoDB11 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 12. DynamoDB ! NoSQL as a Service ! ! SSD 12 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 13. DynamoDB! ! ! 13 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 14. ! SPOF ! 3AZ ! 14 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 15. ! ReadWrite ! ReadDB Read : 1,000 Write : 100! DB Read : 500 Write : 500! DB 15 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 16. ! ! 16 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 17. DynamoDB! Read/Write ! GB http://aws.amazon.com/jp/dynamodb/pricing/ 17 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 18. DynamoDB1. KeyIndex 2. Read/Write Thats it, write your code! 18 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 19. DynamoDB19 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 20. DynamoDB HTTP API Client SideSDKHTTP APIDistributed StorageService SideAPI Servers20 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 21. API ! ! ! ! ! !PutItem UpdateItem GetItem DeleteItem Query Scan! ! ! ! ! !BatchWriteItem BatchGetItem CreateTable DescribeTable UpdateTable DeleteTable21 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 22. API ! ! ! ! ! !PutItem ! BatchWriteItem UpdateItem ! BatchGetItem SDK GetItem ! CreateTable DeleteItem ! DescribeTable Query ! UpdateTable Scan ! DeleteTable22 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 23. Get/Put/Update/Delete/BatchGet Scan Query +23 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 24. AWS SDKs! SDKJavaPythonPHP.NETiOSRubynodeJSAndroid24 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 25. SDK ! Perl Net::Amazon::DynamoDB! Erlang wagerlabs/ddb https://github.com/wagerlabs/ddb! Go go-dynamodb https://github.com/fabiokung/go-dynamodb25 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 26. ! Table Primary KeyHash keyHash key & Range key! Hash key Hash key! Hash key & Range key Hash keyRange key ! Local Secondary Indexes Range key 26 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 27. ! Attributes Hash key, Range keyAttributes Attributes ! Attributes String Number Binary String Set Number Set Binary Set27 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 28. DynamoDB Hash keyHash key & Range key 28 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 29. DynamoDB + 29 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 30. ! Products ID(ProductId)Hash keyProductId (Hash key)ProductName Price1Pen1002Pencil503Eraser10030 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 31. DynamoDB A1-131 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 32. ! IDHash key Primary KeyHashHash key 32 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 33. ! node.jsSDK var aws = require(aws-sdk); aws.config.loadFromPath(PATH/TO/CREDENTIAL); var dynamoDb = new aws.DynamoDB.Client();33dynamoDb.putItem( { TableName:"Products", Item: { ProductId : {N:"1"}, ProductName: {S:"Pen"}, Price: {N:"100"} } }, function(err,data){ if(err){ console.log(err); }else{ console.log(data); } } ); 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 34. ! node.jsSDK dynamoDb.getItem( { TableName:"Products", Key: { ProductId : {N:"1"}, }, function(err,data){ if(err){ console.log(err); }else{ console.log(data); } } );34 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 35. ! Audience ID(AudienceId)Hash keyTimestampRange keyAudienceId (Hash key)ActionUrl12013-10-01 00:01:01Login22013-10-01 00:02:02Login12013-10-01 00:21:00Login135Timestamp (Range key)2013-10-01 00:42:00ViewHoge12013-10-01 00:56:22PostHoge 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 36. DynamoDB ! + 1 2 A2-13 A A1-1 A3-136 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 37. ! 37 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 38. ! node.jsSDK var data = new Array( { AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:01:01" }, Action: { S:"Login" } }, { AudienceId: { N:"2" }, Timestamp: { S:"2013-10-01 00:02:02" }, Action: { S:"Login" } }, { AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:21:00" }, Action: { S:"Login" } }, { AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:42:00" }, Action: { S:"ViewHoge" } }, { AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:56:22" }, Action: { S:"PostHoge" } } );38for (var i =0; i < data.length; i++){ dynamoDb.putItem( { TableName:"Audience", Item: data[i] }, function(err,data){ if(err){ console.log(err); }else{ console.log(data); } } ); 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. } 39. ! node.jsSDK 101 01039dynamoDb.query( { TableName:"Audience", KeyConditions: { "AudienceId": { ComparisonOperator:"EQ", AttributeValueList:[ {N:"1"} ] }, "Timestamp": { ComparisonOperator:"GT", AttributeValueList:[ {S:"2013-10-01 00:10:00"} ] } } }, function(err,data){ if(err){ console.log(err); }else{ console.log(data); } } ); 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 40. LSI ! Local Secondary Indexes(LSI) AudienceId (Hash key)Action Url (Action-Index)12013-10-01 00:01:01Login22013-10-01 00:02:02Login12013-10-01 00:21:00Login140Timestamp (Range key)2013-10-01 00:42:00ViewHoge12013-10-01 00:56:22PostHoge 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 41. Local Secondary IndexesLSI) Get/Put/Update/Delete/BatchGet Scan Query + Local Secondary Index 41 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 42. Local Secondary Indexes (LSI) ! QueryIndex 5ForumLSIKey Query - Replies10 - XX LSI QueryRange Key 42RepliesIndex LastPostIndexLSI 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 43. Local Secondary Indexes ! Secondary Index Query RepliesIndexRepliesIndex Query ForumNameS3 Replies9 ForumName Replies 3 3 Query 43 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 44. LSI Thread LastPostDateTime LSI44LastPostIndexIndex / 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 45. ! Index ThreadRepliesLastPostIndex LastPostIndex 45QueryReplies ReadThroughput 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 46. LSI ! Local Secondary Indexes(LSI) LSI Audience2 Action-IndexLSIAction 46 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 47. LSI ! Action-Index(LSI)47dynamoDb.query( { TableName:"Audience2", IndexName: Action-Index, KeyConditions: { "AudienceId": { ComparisonOperator:"EQ", AttributeValueList:[ {N:"1"} ] }, Action": { ComparisonOperator:EQ", AttributeValueList:[ {S:Login"} ] } } }, function(err,data){ if(err){ console.log(err); }else{ console.log(data); } } ); 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 48. : ! Hash keyRange key Parition1 Partition2 Parition3 1 2 3 4 5 6 7 8 9Range key Partition DynamoDB Partition Parition Hash key Partition 48 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. A,DB,EC,F 49. : ! Hot KeyHash key ! AudienceAudienceId ! Hash key 49 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 50. : LSI ! Local Secondary IndexesIO LSI Write! LSI ! (Projection)Attributes50 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 51. 51 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 52. ! SCANQUERYPutItem 52 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 53. DynamoDB Local ! / DynamoDB CI ! JAR (Java7) ! API ! ! http://bit.ly/1d9fN5c 53 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 54. Transaction Library for DynamoDB ! AWS SDK for Java SDKDynamoDBTransaction Transaction Transaction! ! http://bit.ly/16KbppP54 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 55. Transaction Library for DynamoDB55 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 56. Transaction Library for DynamoDB56 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 57. Amazon EMRHive! hiveExternal Table DynamoDBCREATE EXTERNAL TABLE Audience( AudienceId Int, ActionTimestamp string, Action string ) STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHa ndler' TBLPROPERTIES ("dynamodb.table.name" = Audience2","dynamodb.column.mapping" = AudienceId:AudienceId, ActionTimestamp:Timestamp, Aciton:Action ); 57 http://amazn.to/19goT17 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 58. hiveS3 EMRhiveDynamoDBS3 External Table DynamoDBExternal Table SelectS3External Table Insert INSERT OVERWRITE TABLE s3_as_external_table SELECT * FROM dynamodb_as_external_table;58 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 59. Amazon Redshift ! COPYRedshift COPY audience FROM dynamodb://Audience2 CREDENTIALS 'aws_access_key_id=;aws_secret_access_key=' READRATIO 50; http://amazn.to/19goT17 59 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 60. When to use60 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 61. DynamoDB ! RDB ! DynamoDB ! RDB ! DynamoDB 61 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 62. RDB 62 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 63. MySQL 63 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 64. MySQL 64 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 65. MySQL 65 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 66. MySQL 66 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 67. MySQL 67 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 68. x ) RDB 68 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 69. DynamoDB69 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 70. DynamoDB 70 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 71. DynamoDB ! Hash keyIDRange key ! Hash keyObject S3! IDHash key 71 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 72. 72 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 73. ! SCAN HiveRedshift! NoSQL ()73 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 74. 74 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 75. ! ! ! !DynamoDBNoSQL as a Service NoSQLRDB DynamoDB RDB 75 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 76. ! Amazon DynamoDB http://aws.amazon.com/jp/dynamodb/resources/76 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 77. YahooPeanutsNoSQLDynamo()Cassandra HBaseNoSQL 77 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 78. 78 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.