amazon web services building blocks for drupal applications and hosting

BUILDINGTHECLOUD

JeffBarr,SeniorWebServicesEvangelist

[email protected]

@jeEarr

•  IntroduceyoutoAWS

•  Showyouhowthepiecesfittogether•  Providebackgroundforsecondhalfofthiswebinar•  AnsweryourAWSandcloudcompuRngquesRons

GOALSFORTODAY

•  AWS=“AmazonWebServices”

•  Oneofourthreebusinesses:–  OnlineRetail–  SellerBusiness(Marketplace)

–  WebServiceInfrastructure

•  PayAsYouGoPricingModel•  TwoLevels:

–  InfrastructureasaService(IaaS)–  Pla_ormasaService(PaaS)

INTRODUCINGAWS

Amazon Web Services

Or “AWS” for short…

•  Business

•  Security

•  Presence

•  Technical

BUILDINGTHECLOUD

•  Amazon’sBrandName

•  Self‐serveSignup•  UserAccounts•  PayAsYouGo•  CreditCardBilling

BUILDINGTHECLOUD‐BUSINESS

•  SecurityisJob#1forUs•  DesignedforMulR‐Tenancy•  SAS70CerRficaRon•  SecurityWhitePaper

–  PhysicalSecurity–  LogicalSecurity

•  PublishedBestPracRces•  hfp://aws.amazon.com/security

BUILDINGTHECLOUD‐SECURITY

BUILDINGTHECLOUD‐PRESENCE

Ashburn,VA/Dallas,TX/LosAngeles,CA/Miami,FL/Newark,NJ/PaloAlto,CA/Seafle,WA/St.Louis,MO/Amsterdam/Dublin/Frankfurt/London/HongKong/Tokyo/Singapore/NewYork

USEast(NorthernVirginia)USWest(NorthernCalifornia)Europe(Dublin)AsiaPacific(Singapore)Moretocome…

AWSRegions:

AWSCloudFrontLocaRons

BUILDINGTHECLOUD‐TECHNICAL

•  AmazonEC2:on‐demandcomputepower–  Obtainandbootnewserverinstancesinminutes–  Scalecapacityupordownasneeded–  $0.085perhourandup

•  Keyfeatures:–  SupportforWindows,Linux,andOpenSolaris–  Completeflexibilitytoinstallanyopensourceorcommercialsoqware–  SupportsallmajorwebandapplicaRonpla_orms–  MulRpleAvailabilityZones–  ElasRcIPAddresses–  ElasRcBlockStorage–  ElasRcLoadBalancing–  AutoScaling–  AmazonCloudWatch

AMAZONELASTICCOMPUTECLOUD

•  CreateanewrelaRonaldatabasewithoneAPIcall•  OffloadcommonadministraRvetaskstoAWS•  UseexisRngcodeandtools•  ScaleupprocessingorstoragewithoneAPIcall

•  Recoverdatabaseusingsnapshotorlogfile

•  MySQL5.1

AMAZONRELATIONALDATABASESERVICE(RDS)

2010Jan Feb Mar Apr May Jun Jul

»AmazonEC2ReservedInstanceswithWindows,ExtraLargeHighMemoryInstances»AmazonS3VersioningFeature»ConsolidatedBillingforAWS»LowerpricingforOutboundDataTransfer

»AmazonSNS

»SingaporeRegion»RDSinEurope»ELBsessions

AWSPACEOFINNOVATION–2010

»RDSConsoleSupport» S3ReducedRedundancyStorage» RDSMulR‐AZ» CloudFrontStreamingLogs» VPCinEurope

»AWSSoluRonPages

»RDSSSLSupport»CloudWatchEBSMetrics»S3ConsoleSupport»Import/ExportAPI»CloudFrontHTTPSSupport»CloudFrontPOPinNY»CloudFrontPriceReducRon»ElasRcMapReduceupdates

»??»??»??

•  AWSTrends:–  ConRnuousinnovaRon–  AddiRonalservices–  AddiRonalfeatures–  AddiRonallocaRons

•  ThisistheRmetolearnandstarttousethecloud!

FINALTHOUGHTS

•  Createdeveloperaccountathfp://aws.amazon.com

•  Downloadaccesskeys,SDKs,developertools

•  BuildanddeployapplicaRon

•  AWSBlog:hfp://aws.typepad.com

GETTINGSTARTEDWITHAWS

© 2010 Acquia, Inc. Amazon Web Services Building Blocks for Drupal Applications and Hosting

Hosting Drupal on AWS

Barry Jaspan Senior Architect, Acquia [email protected] @bjaspan

http://acquia.com/resources/acquia-tv


Acquia Hosting Overview •  Managed web hosting for Drupal

•  High performance, scalability, availability

•  Combined infrastructure- and application-level support from one vendor

•  Single or multi-tier •  Load balancers, web nodes, database clusters,

memcached, network filesystems, ...

•  One instance, separate instances, or any combination

•  Automated •  We provisioned for a “20M uniques on day one”

customer ... in a few minutes.


Load balancing


Load balancing on AWS •  No hardware LB is available

•  Round-robin DNS to web nodes is bad for high-availability

•  Elastic Load Balancer (ELB) •  CNAME requires http://www.TLD

•  Limited flexibility (but now with stickiness!)

•  Elastic IP address (EIP) + software LB •  Allows http://TLD

•  Only one EIP per server

•  We use nginx for LB and Varnish for caching


Software load balancer


Shared !les


File sharing on AWS •  Drupal needs a POSIX filesystem for user-

uploaded files

•  Elastic Block Store (EBS) is single-instance

•  rsync does not scale past N=2 web nodes

•  S3FS is worse than useless for this •  Too slow, non-POSIX, unreliable, unsupported

•  A network filesystem is required •  NFS?

•  More later...


Basic web stack


HA load balancer •  Load balancer might fail

•  Elastic IP •  Requires a hot spare

•  EIP must be reassigned to a hot spare

•  Elastic Load Balancer •  For us, can’t talk directly to web nodes

•  nginx provides virtual host flexibility, concurrency protection, etc

•  Varnish provides caching

•  Also requires a hot spare (or second active instance)


HA load balancer with EIP


HA !lesystem •  NFS

•  Cannot tolerate server failure

•  Not replicated •  DRBD with virtual IP reassignment won’t work

•  AFS •  Tolerates server failure

•  Not replicated

•  GlusterFS •  Replicated and distributed

•  GPL, commercially supported, under active dev

•  Easy to use, tricky to optimize just right


HA !lesystem with GlusterFS


HA database •  Multi-AZ Relational DB Service (RDS)

•  Failover takes 3 minutes

•  Active/passive master MySQL replication •  Must avoid “split-brain” operation

•  All web nodes must fail over simultaneously

•  Heartbeat with virtual IP reassignment not possible

•  Failover mechanism must itself be HA •  Using MySQL proxy just leads to the same problem

•  Requires synchronous app-level failover

•  We store db election in GlusterFS •  Any other HA replicated data store would do


High-availability web stack


Machine con!g •  All these instances need to be built

•  Incremental custom AMIs •  Time consuming during development/debugging

•  Poor fit for source-code control

•  Leads to mystery configurations

•  Automated configuration scripts/language •  Incremental updates are simple

•  Good fit for source-code control

•  Known process from a clean, base AMI

•  Combining the two may be best


Static con!g with puppet


Elasticity •  Elastic scalability means dynamic

configuration •  Balancers: info on all web nodes

•  Web nodes (Drupal): info on all db servers

•  FS servers: info on all EBS volumes to use

•  FS clients: info on all FS servers to use

•  All (re-)started instances get a new IP •  EIPs are limited and cost more to use

•  All servers need current internal IPs

•  Admins need current external IPs

•  Requires frequent manual updates, or automation


Dynamic data-driven con!g


Email from AWS •  AWS IPs are on many spam blacklists

•  Can’t reliably be used to send mail

•  Requires a non-cloud mail server or service provider

•  Note: Jeff tells me this situation has improved


Colo mail server


Everything else •  I’ve only talked about cloud-specific issues

•  To self-host (anywhere), you also need: •  Backups

•  24x7 monitoring

•  Ops staff

•  Caffeine

•  Ibuprofen


Management and ops stack

amazon web services building blocks for drupal applications and hosting

Technology

elastic load

load balancing

hot spare

web nodes

drupal applications

aws

requires

drupal