20131107 cwt2013-wdkz

41
AmebaにおけるRHadoopの活用事例 株式会社サイバーエージェント アメーバ事業本部 Ameba Technology Laboratory 和田 計也

Upload: cyberagent

Post on 08-Jul-2015

481 views

Category:

Documents


4 download

TRANSCRIPT

  • 1. AmebaRHadoop Ameba Technology Laboratory

2. AmebaAmeba Technology Laboratory Patriot RHadoop2 3. Ameba Ameba Technology Laboratory 4. Ameba PC4 5. Ameba 5 6. Ameba 6 7. Ameba 7 8. Ameba Technology Laboratory Ameba 2011 ()8 9. Patriot 10. 10AmebaPatriot Ameba Hadoop Hive/HBase Hive Flume90,000lines/sec1TB/day11,000jobs/day 11. Log SCP MySQL HiveAmeba FlumeHiveJob Batch Put HBase Hadoop View Hive WebUI Hive 11 12. Patriot WebView / 12 13. PatriotCDH 2010 Patriot (CDH3b) 2011 CDH3u0 Patriot (CDH3u3) PatriotDCCDH(CDH4.3)13 14. RHadoop RHadoopRandomForest RHadoop 15. 15R 1993 Version1.02000 201311Version 3.0.2Ross IhakaRobert Gentleman 16. 16R http://r4stats.com/articles/popularity/ 17. 17RHadoop n RHadoopR n rmr n rhdfs n rhbase n plyrmrn Revolution Analytics OSS nhttps://github.com/RevolutionAnalytics/RHadoop/wiki 18. 18RHadoopClouderaRevolution Analytics http://www.cloudera.com/content/cloudera/en/solutions/partner/Revolution-analytics.html 19. RHadoop RHadoopRandomForest RHadoop 20. 20RandomForest n n n Tree n n http://opinions5.blogspot.jp/2013/08/random-forest-confidence.html 21. 21RandomForest n n n n n web n MahoutDecisionForest . 22. RHadoopHadoop RandomForest(model) train model model model model Map Reduce22 23. RHadoopHadoop RandomForest(predict) test(block)mod models mod els els MapReduce 23 24. 24) # source(R/scaleRandomForest.R) #weight srf_midub