time series analysis by javascript ll matsuri 2013

Time Series analysis by JavaScript

LL matsuri 2013/08/24

1

Time Series analysis by JavaScript

前処理

LL matsuri 2013/08/24

2

{twitter: “muddydixon”}✓Data Mining✓Data Visualization✓Love D3.js

✓working @ NIFTY

3

NIFTY Cloud

4

Visualization✓What is Visualization?

communicating informationclearly and effectively

5

Visualization on Web✓Profit✓Watch KGI / KPI continuously✓Share visualization via URL

✓empowered representation by HTML5 / css3

-> D3.js6

Visualization on Web : D3.js

7

Time Series Visualization✓Visualize “time series” of ✓KGI / KP of services✓Metrics of system resources

8

Sample{"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427}{"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494}{"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888}{"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641}{"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452}{"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346}{"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358}{"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969}{"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213}{"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109}{"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633}{"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777}{"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732}{"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024}{"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203}{"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324}{"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145}{"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285}{"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979}{"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581}{"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261}{"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419}{"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493}{"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272}{"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287}{"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856}{"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998}{"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702}{"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021}{"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679}{"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319}{"time":"2013-01-02T08:59:43.000Z","cpu":0.6149,"memory":0.6988}{"time":"2013-01-02T09:00:40.000Z","cpu":0.4249,"memory":0.6441}{"time":"2013-01-02T09:00:52.000Z","cpu":0.5086,"memory":0.5189}{"time":"2013-01-02T09:00:02.000Z","cpu":0.3182,"memory":0.4765}{"time":"2013-01-02T09:00:02.000Z","cpu":0.4201,"memory":0.4376}{"time":"2013-01-02T09:01:20.000Z","cpu":0.4181,"memory":0.5856}{"time":"2013-01-02T09:01:46.000Z","cpu":0.6682,"memory":0.5992}{"time":"2013-01-02T09:01:22.000Z","cpu":0.3787,"memory":0.4777}{"time":"2013-01-02T09:01:54.000Z","cpu":0.6057,"memory":0.3374}{"time":"2013-01-02T09:01:42.000Z","cpu":0.3485,"memory":0.6665}{"time":"2013-01-02T09:01:45.000Z","cpu":0.3628,"memory":0.6092}{"time":"2013-01-02T09:01:47.000Z","cpu":0.6096,"memory":0.5961}{"time":"2013-01-02T09:01:02.000Z","cpu":0.5355,"memory":0.6442}

9

可視化エンジニアマネージャー

デイリーで売上データ見せて

了解しました

10

可視化エンジニアマネージャー

やっぱり、月次と週次で

…了解しました

11

可視化エンジニアマネージャー

あと売上の平均と分散も

うっ、つらい

12

Summarizekeyvalues = {}for d in data time = new Date(d.time.getFullYear(), d.time.getMonth(), d.time.getDate(), d.time.getHours(), d.time.getMinutes()) keyvalues[time] = [] unless keyvalues[time] keyvalues[time].push d

aggregated = []for time, values of keyvalues obj = {time: time, cpu: 0, memory: 0} for d in values obj.cpu += d.cpu obj.memory += d.memory aggregated.push obj

console.log aggregated

13

ProblemsLarge

Time perspectives / Statistics Avoid to get data each time

Hourly / Daily / Weekly / Monthly / YearlySummary / Average / Stdev

14

Problems

メンドクサイ

15

Series.js

16

Series.js✓Providing utility methods for time series✓sum, sum square, mean, median, quantile✓variance, standard devience, covariance, correlation✓auto covariance, auto correlation✓Providing aggregation✓minutes, hour, day, week, month, year✓Coming soon✓auto regression, spectrum✓moving average, holt-winters, exponential moving average

17

Series.js APIs example# class methodsSeries.sum([1,2,3,4,5]) #=> 15Series.mean([1,2,3,4,5]) #=> 3

# instance methodss = new Series()s.put d for d in [1,2,3,4,5]s.mean() #=> 3

18

Series.js APIs example# accessorSeries.y((d)-> d.v).sum([{v:1},{v:2},{v:3},{v:4},{v:5}]) #=> 15s = new Series().y((d)-> d.v)s.put d for d in [{v:1}, {v:2}, {v:3}, {v:4}, {v:5}]s.mean() #=> 3

19

Series.js APIs example# aggregationdata = [{t: 1, v:1}, {t: 1, v:2}, {t:2, v:3}, {t:2, v:4}, {t:1, v:5}]Series.y((d)-> d.v).aggregation(Series.sum).key((d)-> d.t)(data)#=> [{t:1, v: 8}, {t: 2, v: 7}]

20

Sample{"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427}{"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494}{"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888}{"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641}{"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452}{"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346}{"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358}{"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969}{"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213}{"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109}{"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633}{"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777}{"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732}{"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024}{"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203}{"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324}{"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145}{"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285}{"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979}{"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581}{"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261}{"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419}{"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493}{"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272}{"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287}{"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856}{"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998}{"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702}{"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021}{"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679}{"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319}{"time":"2013-01-02T08:59:43.000Z","cpu":0.6149,"memory":0.6988}{"time":"2013-01-02T09:00:40.000Z","cpu":0.4249,"memory":0.6441}{"time":"2013-01-02T09:00:52.000Z","cpu":0.5086,"memory":0.5189}{"time":"2013-01-02T09:00:02.000Z","cpu":0.3182,"memory":0.4765}{"time":"2013-01-02T09:00:02.000Z","cpu":0.4201,"memory":0.4376}{"time":"2013-01-02T09:01:20.000Z","cpu":0.4181,"memory":0.5856}{"time":"2013-01-02T09:01:46.000Z","cpu":0.6682,"memory":0.5992}{"time":"2013-01-02T09:01:22.000Z","cpu":0.3787,"memory":0.4777}{"time":"2013-01-02T09:01:54.000Z","cpu":0.6057,"memory":0.3374}{"time":"2013-01-02T09:01:42.000Z","cpu":0.3485,"memory":0.6665}{"time":"2013-01-02T09:01:45.000Z","cpu":0.3628,"memory":0.6092}{"time":"2013-01-02T09:01:47.000Z","cpu":0.6096,"memory":0.5961}{"time":"2013-01-02T09:01:02.000Z","cpu":0.5355,"memory":0.6442}

21

Summarize with Series.jsSeries.y((d)-> d.cpu).aggregation(Series.sum) .minute((d)-> d.time)(data)#=> [{ t: 'Sun Jan 13 2013 23:29:00 GMT+0900 (JST)', y: 2.0095 }, { t: 'Sun Jan 13 2013 23:30:00 GMT+0900 (JST)', y: 3.4370 }, { t: 'Sun Jan 13 2013 23:31:00 GMT+0900 (JST)', y: 1.58180 }, { t: 'Sun Jan 13 2013 23:32:00 GMT+0900 (JST)', y: 2.2742 }, { t: 'Sun Jan 13 2013 23:33:00 GMT+0900 (JST)', y: 3.4541 }, { t: 'Sun Jan 13 2013 23:34:00 GMT+0900 (JST)', y: 4.6035 }, { t: 'Sun Jan 13 2013 23:35:00 GMT+0900 (JST)', y: 2.1165 }, { t: 'Sun Jan 13 2013 23:36:00 GMT+0900 (JST)', y: 1.6177 },

22

TODO

PerformanceRefactoringDocument

23

TODO

Client side

24

TODO

Client sideさっきpushしました

25

CROSS 2014

26

以下、ビールを飲んで笑顔な技術者の顔が続いていると思ってください

27