1. 1. wemakeprice . Facebook DevOps Korea 2015 6 8() (chingu94@gmail.com) Facebook, Flickr, Netflix, Etsy DevOps and Continuous Deployment
2. 2. Agenda 2 3 1
3. 3. 1. DevOps ? 3 .
4. 4. 1. DevOps ? 4 5 KPCB INTERNET TRENDS 2015 : http://www.kpcb.com/internet-trends
5. 5. 1. DevOps ? 5 : http://www.kpcb.com/internet-trend s 1995 2014 Internet Users 0.6% (35MM+) 39% (2.8B) Mobile Phone Users (Feature Phone ) 1% (80MM+) 73% (5.2B) 1995
6. 6. 1. DevOps ? 6 : http://www.kpcb.com/internet-trend s 10
7. 7. 1. DevOps ? 7 : http://www.kpcb.com/internet-trend s , 2000 ~ 2015 (15)
8. 8. 1. DevOps ? 8 : http://www.kpcb.com/internet-trend s
9. 9. 1. DevOps ? 9 , , .
10. 10. 1. DevOps ? 10 (http://www.reactivemanifesto.org/)Reactive Manifesto These changes are happening because application requirements have changed dramatically in recent years. Only a few years ago a large application had tens of servers, seconds of response time, hours of offline maintenance and gigabytes of data. Today applications are deployed on everything from mobile devices to cloud-based clusters running thousands of multi-core processors. Users expect millisecond response times and 100% uptime. Data is measured in Petabytes. Few years ago Today Large application? Tens of servers Thousands of multi-core proc essors Seconds of response time Millisecond response time Hours of offline maintenance 100% uptime Gigabytes of data Petabytes of data Today's demands are simply not met by yesterdays software architectures.
11. 11. 1. DevOps ? 11 , . . , Rush . ,
12. 12. 1. DevOps ? 12 , ?
13. 13. 1. DevOps ? 13 The Hacker Way Facebook IPO Focus on Impact Move Fast Be Bold Be Open Build Social Value . (Done is better than perfect.) . (Code wins arguments.) . : http://www.looah.com/article/view/738/
14. 14. 1. DevOps ? 14 : http://www.google.com/intl/ko_KR/about/company/philosophy/ Ten Things Google Knows To Be True Focus on the user and all else will follow Its best to do one thing really, really well. Fast is better than slow. Democracy on the web works. You dont need to be on your desk to need an answer. You can make money without doing evil. There is always more information out there. The need for information crosses all border. You can be serious without a suit. Great just isnt good enough.
15. 15. 1. DevOps ? 15 : http://www.slideshare.net/diannemarsh/from-code-to-the-monkeys-continuous-delivery-at- netflix/ Amazon Leadership Principles 1. Customer Obsession 2. Ownership 3. Invent and Simplify : http://www.amazon.jobs/principles/ Netflix Goal Move Fast, Fail Fast (and Small) 1. Dianne Marsh (Director of Engineering for Cloud Tools at Netflix)
16. 16. 1. DevOps ? 16 Moving Fast!! Focus on Customer!!
17. 17. 1. DevOps ? 17 .
18. 18. 1. DevOps ? 18 , DevOps ? .
19. 19. 1. DevOps ? 19 DevOps (Wikipedia) DevOps , . . : http://ko.wikipedia.org/wiki/%EB%8D%B0%EB%B8%8C%EC%98%B5%EC%8A%A4
20. 20. 1. DevOps ? 20 Company Deploy Frequency Deploy Lead Time Reliability Customer Responsiveness Amazon 23,000 / day Minutes High High Google 5,500 / day Minutes High High Netflix 500 / day Minutes High High Facebook 1 / day Hours High High Twitter 3 / week Hours High High Typical enterprise Once every 9 months Months or quarters Low / Medium Low / Medium : The Phoenix Project , Amazon, Google, Netflix, Facebook, Twitter ?
21. 21. 1. DevOps ? 21 DevOps ?
22. 22. 2. FB, Flickr, Netflix, Etsy DevOps ? 22 ? 1. Ars technica : http://arstechnica.com/business/2012/04/exclusive-a-behind-the- scenes-look-at-facebook-release-engineering/ : http://swprocess.egloos.com/3009704 The Hacker Way: http://www.looah.com/article/view/738 Robert Johnson InfoQ : http://www.infoq.com/presentations/Facebook-Moving-Fast-at-Scale
23. 23. 2. FB, Flickr, Netflix, Etsy DevOps ? 23 2. ( ) 3. Deployment Pipeline : http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6449236
24. 24. 2. FB, Flickr, Netflix, Etsy DevOps ? 24 4. HipHop PHP -> C++ . 1.5 GB. 1.5GB push BitTorrent BitTorrent (rack) 30 ( : 15, : 15, 2012 ) . [Facebook . : http://www.looah.com/article/view/983]
25. 25. 2. FB, Flickr, Netflix, Etsy DevOps ? 25 5. FB Single stable branch , long-lived branche 6. Tools : Phabricator (http://phabricator.org/) : Watir (http://watir.com/) : Selenium (https://github.com/seleniumhq/selenium) : Perflab 7. IRC IRC ( 700) 8. , , Facebook
26. 26. 2. FB, Flickr, Netflix, Etsy DevOps ? 26 9. . . . , . (/ ) . Facebook Hotfix Bar . . . : http://www.looah.com/article/view/983
27. 27. 2. FB, Flickr, Netflix, Etsy DevOps ? 27 1. 10 deploys per day Flickr John Allspaw(Ops head) Paul Hammond(Dev head) Velocity 2009 : : https://www.youtube.com/watch?v=LdOe18KhtT4
28. 28. 2. FB, Flickr, Netflix, Etsy DevOps ? 28 2. [ ] Dev [ ] Ops !! Ops and Devs job is to enable the business. The business requires change. ? ? . Tools Culture : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
29. 29. 2. FB, Flickr, Netflix, Etsy DevOps ? 29 3. Tools Automated infrastructure A. Flickr Tools: Chef, Cfengine, Puppet, FAI, Cobbler Shared version control A. Dev/Ops version control One step build and deploy A. B. , , : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
30. 30. 2. FB, Flickr, Netflix, Etsy DevOps ? 30 : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
31. 31. 2. FB, Flickr, Netflix, Etsy DevOps ? 31 3. Tools Feature flags: on/off Trunk Always ship trunk Bucket testing Dark launches Shared metrics IRC and IM robots Dev, Ops IRC IM robots build, deploy, alerts monitors : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
32. 32. 2. FB, Flickr, Netflix, Etsy DevOps ? 32 4. Culture Respect: , . Trust: Healthy attitude about failure Avoiding Blame , !! : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
33. 33. 2. FB, Flickr, Netflix, Etsy DevOps ? 33 Etsy: , , , 2008 (40+ engineers) Painful merge Hand off to deployers Deploy, site down Roll back deploy Fix bugs, go to step 2 6? 25+ . 2014 (400+ engineers) Small changesets, deployed frequently Engineers deploy (not just engineers, but also Designers, Dogs) Deploys are fast Changes behind flags Copious graphs/metrics Fix fast & roll forward (http://www.slideshare.net/beamrider9/continuous-deployment-at-etsy-a-tale-of-two-approaches)
34. 34. 2. FB, Flickr, Netflix, Etsy DevOps ? 34 1. Netflix API (Tech Blog): http://techblog.netflix.com/2013/08/deploying-netflix-api.html Self service build and deployment at Netflix: http://www.slideshare.net/garethbowles/self- servicebuilddeploymentagile2013?related=2 Dianne Marsh (Director of Engineering for Cloud Tools): http://www.slideshare.net/diannemarsh/from-code-to-the-monkeys- continuous-delivery-at-netflix?related=2 Continuous Delivery at Netflix: http://www.slideshare.net/robspieldenner/continuous-delivery-at- netflix?related=1 : http://techblog.netflix.com/2013/08/deploying-netflix-api.html
35. 35. 2. FB, Flickr, Netflix, Etsy DevOps ? 35 2. Deployment Flow Unit Test Regression Test Canary Analysis Red/Black Global Release (Red) : http://techblog.netflix.com/2013/08/deploying-netflix-api.html
36. 36. 2. FB, Flickr, Netflix, Etsy DevOps ? 36 3. Branches 3 long-lived branches (Test/Release/Prod branch) Test branch: test branch deploy production release branch merge Release branch: weekly release. Release branch commit production staging deploy . Staging production deploy delivery team . Prod branch: Release release branch prod branch merge. Prod branch Patch/Daily push . production deploy weekly release deploy prod branch commit. Prod branch commit release branch merge live portion canary cluster deploy. Canary analysis go global deploy : http://techblog.netflix.com/2013/08/deploying-netflix-api.html
37. 37. 2. FB, Flickr, Netflix, Etsy DevOps ? 37 4. Canary Canary? production small subset Canary (HTTP status code, response time, , load avg ) 1000 metric 100 . : http://techblog.netflix.com/2013/08/deploying-netflix-api.html
38. 38. 2. FB, Flickr, Netflix, Etsy DevOps ? 38 5. Region Netflix 3 AWS region (2013 ) API deploy Asgard (https://github.com/Netflix/asgard) Red/Black push production push A. AWS Auto Scale Group( ASG) Red/Black push . B. ASG 10% ASG . . (red/red ) C. ASG . (red/black ) D. ASG . ASG 6. production push XMPP bot : http://techblog.netflix.com/2013/08/deploying-netflix-api.html
39. 39. 2. FB, Flickr, Netflix, Etsy DevOps ? 39 5. Region Netflix 3 AWS region (2013 ) API deploy Asgard (https://github.com/Netflix/asgard) Red/Black push production push A. AWS Auto Scale Group( ASG) Red/Black push . B. ASG 10% ASG . . (red/red ) C. ASG . (red/black ) D. ASG . ASG 6. production push XMPP bot
40. 40. 3. ? 40 ? ?
41. 41. 3. ? 41 1. & Lean Lean Lean (Lean Thinking) S/W . , Lean . : http://zzino.co.kr/blog/?p=173
42. 42. 3. ? 42 1. , (Lean ) Lean Software Development Principles 1. Eliminate waste ( ) 2. Amplify learning ( ) 3. Decide as late as possible ( ) 4. Deliver as fast as possible ( delivery ) 5. Empower the team ( ) 6. Build quality in (quality ) 7. See the whole ( ) : http://www.slideshare.net/jpvajda/lean-software-development-principles
43. 43. 3. ? 43 2. : Lean Enterprise
44. 44. 3. ? 44 3. ? Continuous Integration > Continuous Deployment > DevOps ...
45. 45. 3. ? 45 4. . DevOps . 10+ Deploys Per Day Flickr John Allspaw SVP Etsy Continuous Deployment 6 . ( p33 ) DevOps , , , .
46. 46. 3. ? 46 5. . : Lean Enterprise
47. 47. 47
48. 48. References 48 KPCB : http://www.kpcb.com/internet-trends Reactive Manifesto: http://www.reactivemanifesto.org/ Facebook ( ): http://www.looah.com/article/view/983/ Development and Deployment at Facebook: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6449236 [ ] ?: http://swprocess.egloos.com/3009704/ InfoQ Facebook Moving Fast at Scale : http://www.infoq.com/presentations/Facebook-Movin g-Fast-at-Scale 10+ Deploys Per Day at Flickr : https://www.youtube.com/watch?v=LdOe18KhtT4 10+ Deploys Per Day at Flickr : http://www.slideshare.net/jallspaw/10-deploys-per-day-dev -and-ops-cooperation-at-flickr/ Continuous Deployment at Etsy : http://www.slideshare.net/beamrider9/continuous- deployment-at-etsy-a-tale-of-two-approaches Cooperation Collaboration Awareness at Etsy & Flickr: http://www.slideshare.net/jallspaw/dev-and- ops-collaboration-and-awareness-at-etsy-and-flickr Netflix Culture: Freedom & Responsibility : http://www.slideshare.net/reed2001/culture- 1798664?related=1 Netflix API deploy ( ): https://josephkim75.wordpress.com/2013/10/23/netflix%EC%97%90%EC%84%9C-api- deploy%ED%95%98%EA%B8%B0/
49. 49. References 49 Principles and Practices in Continuous Deployment at Etsy: http://www.slideshare.net/mikebrittain/p rinciples-and-practices-in-continuous-deployment-at-etsy?related=3 Continuously Deploying Culture (Scaling Culture at Etsy) : http://www.slideshare.net/mcdonnps/continuously-deploying-culture-scaling-culture-at-etsy- 14588485?related=4
50. 50. Further Reading 50