使用amazon kinesis firehose 和 用amazon... · pdf file• 异步调用lambda. aws...
If you can't read please download the document
TRANSCRIPT
AWS
AWS
Amy Li, Technical Trainer, Amazon Web Services
Amazon Kinesis Firehose Amazon Redshift
AWS
AWS
Amazon Kinesis Firehose & Amazon Redshift
Step 1: Redshift
Step 2: Firehose Delivery Stream
Step 3: Firehose Delivery Stream
Step 4: Redshift
Step 5:
Q & A
AWS
AWS
Volume
Velocity
Variety
Value
AWS
()-
"Pipeline"
AWS
AWS
Business Intelligence and Data
Visualization
Amazon QuickSight
Elastic Search Analytics
Amazon Elasticsearch Service
Hadoop Ecosystem
Amazon EMR
Near Real-time
AWS Lambda
Amazon Kinesis Analytics
Data Warehousing
Amazon Redshift
Machine Learning
Amazon Machine Learning
Elastic Search Analytics
Amazon Elasticsearch Service
Process and Move Data
AWS Data Pipeline
Ad Hoc Analytics
Amazon Athena
Object Storage
Amazon S3
Amazon Glacier
Near Real-time
Amazon Kinesis Streams
RDBMS
Amazon RDS
NoSQL
DynamoDB
Search
Amazon CloudSearch
Internet of Things (IoT)
Amazon IoT
Near Real-time
Amazon Kinesis Firehose
Data Import
Amazon Import/Export Snowball
Message Queuing
Amazon SQS
Web/app Servers
Amazon EC2
AWS
?
AWS
IoT
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test
AWS
(M. Gualtieri, Forrester)
+
AWS
AWS
Amazon Kinesis
TB
AWS
Amazon Kinesis Streams
AWS
Amazon Kinesis Streams
:
: Kinesis Client Library (KCL), Apache Spark/Storm, AWS Lambda, ....
:
AWS
Data
Sources
App.4
[Machine
Learning]
AW
S E
nd
po
int
App.1
[Aggregate &
De-Duplicate]
Data
Sources
Data
Sources
Data
Sources
App.2
[Metric
Extraction]
App.3[Sliding
Window
Analysis]
Data
Sources
Availability
Zone
Shard 1
Shard 2
Shard N
Availability
ZoneAvailability
Zone
Amazon Kinesis
AWS
Amazon Kinesis Firehose
AWS
Amazon Kinesis Firehose
Data sources
AW
S en
dp
oin
t
Data sources
Data sources
Data sources
Data sources
No Partition Keys
No Provisioning
End-to-End Elastic
Amazon S3
Amazon Redshift
Amazon
Elasticsearch
Service
AWS
Kinesis Firehose
Delivery StreamKinesis Firehose
delivery stream Kinesis Firehose
Record Producer delivery stream
Kinesis Firehose
Buffer Size
MBBuffer Interval 1000 KB
AWS
Firehose S3
AWS
Firehose Redshift
AWS
Firehose Elasticsearch
AWS
Kinesis Firehose:
AWS SDK
PutRecord()
PutRecordBatch()
Kinesis Agent
Firehose delivery stream Amazon CloudWatch
AWS
Kinesis Firehose
Kinesis Firehose AWS Lambda
Lambda
AWS
Amazon Redshift
PB
MPP
AWS
AWS
Step 1 Redshift
AWS
Redshift
AWS
AWS
Step 2: Firehose Delivery Stream
AWS
Step 3: Firehose Delivery Stream
AWS
Sample Data
219.134.32.117 - - [16/Feb/2017:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521
"-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.1; .NET CLR
3.8.23015.5)"
95.169.41.62 - - [16/Feb/2017:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200
3883 "-" "Mozilla/5.0 (Windows NT 6.2; Trident/7.0; rv:11.0) like Gecko"
221.147.191.247 - - [16/Feb/2017:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko)
Chrome/38.0.895.0 Safari/538.0.1"
179.96.123.130 - - [16/Feb/2017:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-"
"Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.4) Gecko/20100101 Firefox/5.4.6"
132.119.12.76 - - [16/Feb/2017:09:38:20 -0800] "PUT /explore HTTP/1.1" 200 3131 "-"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_0 rv:5.0; AZ) AppleWebKit/535.1.0
(KHTML, like Gecko) Version/4.0.3 Safari/535.1.0"
74.113.56.92 - - [16/Feb/2017:09:38:20 -0800] "DELETE /app/main/posts HTTP/1.1" 200
7069 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_9) AppleWebKit/532.1.0 (KHTML,
like Gecko) Chrome/15.0.877.0 Safari/532.1.0"
AWS
After Data Transformation
{"host":"26.56.11.130","ident":"-","authuser":"-","request":"GET /wp-content
HTTP/1.1","response":200,"bytes":4582,"verb":"GET","@timestamp":"2017-04-
04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"}
{"host":"180.153.215.216","ident":"-","authuser":"-","request":"PUT /search/tag/list
HTTP/1.1","response":200,"bytes":1461,"verb":"PUT","@timestamp":"2017-04-
04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"}
{"host":"155.233.163.37","ident":"-","authuser":"-","request":"GET /explore
HTTP/1.1","response":500,"bytes":326,"verb":"GET","@timestamp":"2017-04-
04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"}
{"host":"189.176.106.5","ident":"-","authuser":"-","request":"POST /search/tag/list
HTTP/1.1","response":200,"bytes":3059,"verb":"POST","@timestamp":"2017-04-
04T11:32:29.000Z","timezone":"-0700","@timestamp_utc":"2017-04-04T18:32:29.000Z"}
AWS
Step 4: Redshift
Step 5:
AWS
AWS
Amazon Kinesis Firehose & Amazon Redshift
Apache
?
AWS
AWS
AWS
aws.amazon.com/training
aws.amazon.com/certification
AWS-
https://aws.amazon.com/cn/training/https://aws.amazon.com/cn/certification/mailto:[email protected]