introduction to hortonworks data cloud for aws

1 ©HortonworksInc.2011– 2016.AllRightsReserved

HortonworksDataCloudEnterprisereadyHadooponthecloud

蒋逸峰（しょういつほう／YifengJiang）SolutionsEngineer,Hortonworks@uprushDecember14,2016


About Me

蒋逸峰 (しょういつほう / Yifeng Jiang)• Solutions Engineer, Hortonworks• Apache HBase book author• I like hiking & running• Twitter: @uprush


HortonworksDataPlatform(HDP)


What’sMissing?

Ã Ambari makesdeployingHDPsupereasy,but..– Itisnoteasytogetthere– Clustersizing– HWpurchase,setupinDC,network– OSsetup

Ã Averagethreeweeksorevenmore

©HortonworksInc.2011– 2016.AllRightsReserved6

IntroducingHortonworksDataCloudforAWS

Ã AnewcloudproductfromHortonworks– PoweredbyHortonworks DataPlatform

Ã OffersPay-As-You-Go(PAYG)pricing

Ã DeliveredandsoldviaAWSMarketplace

Ã HandlesmostcommonbigdatausecaseswithApacheHadoop,Spark,andHive– Choosefromasetofprescriptiveclustertypes

Ã Focusesoneaseofuseandbusinessagility– Avoidsinfiniteconfigurabilityandcustomization

Ã OptionalFreeCommunitySupport**

**EnterpriseSupportoptioncomingsoon


DEMO


Architecture

AmazonWebServices

CloudbreakServices

Cloudcontroller(akaCloudbreak)

CloudbreakDB

Connector

AWS GCE Azure

HDPCluster:ETL/EDW

MasterGroupMasterGroup:Hive,Spark

Ambari

SlaveGroup

Blueprint

HDPCluster:Analytics

MasterGroupMasterGroup:LLAP,Zeppelin

Ambari

SlaveGroup

Blueprint

CloudbreakDeployer

Accesstools

Shell RESTAPI WebUI

OpenStack

S3aFileSystem

S3aFileSystem


HortonworksDataCloud- Summary

Ã Launchandmanageclustersbyworkloadtype– ETL/EDW,Datascience,Businessanalytics

Ã Usehighlyscalable,durablestoragefordata(S3)&metadata(RDS)

Ã Sharedataandmetadataamongmultipleephemeralclusters

Ã Scaleupanddownattheclickofabutton

Ã SecureclusterswithIAMroles,securitygroups,etc.


ImprovingEnterpriseReadiness


EnterpriseReadiness

ImprovingenterprisereadinessinthecloudÃ CloudstorageÃ SecurityandgovernanceÃ Reliabilityandfaulttolerance


MatchingHadoopwiththeCloud

Datacenter• DataLocality• Consistent

Storage• Singlecluster

administration

Cloud• Scalablestorage• Customizability• Costeffective

compute

• Scalablestoragewithperformanceandconsistency

• Customizabilitywitheaseofadministration

• Costeffectivecomputewith SLApolicies


CloudStorageaccessfacts

HDFS

Application

Input Output tmp

Interactionmodels

Application

HDFSInput

Output

Copy

Ã Cloudstorageoptimizesforscale– S3dataisreplicatedforhighscaleaccess,durability

Ã Dataaccessisremote– Datalocality– Costliermetadataoperations(E.g.hadoop fs–mv isactuallyacopyanddelete)

Ã EventualConsistency– Takestimeforeffectofmodificationoperationstopermeatetoallcopies


PerformancewithScalability

Ã Generalstrategy:Optimizebyworkloadtypes

Ã ETLworkloads

– Typicalpipeline:Bringindata=>Transform=>Repairpartitions=>Computestatistics

– Multiplemetadatacalls:Batchedandissuedinparallelforperformancegains

Ã Distcp

– Optimizedbuffermanagementfortransferringlargefiles

– RandomizeinputtoDistcp toavoidhot-spottingS3nodes


PerformancewithScalability

Ã Analyticsworkloads– ORCfilerelatedoptimizations

– Supportfastrandomaccessreads(bothdirections)byavoidingtearingdown

S3HTTPconnections

– Passindexinformationtocomputetasksaspartofsplitdatatoavoidre-

computation

Ã Status:Available,butperformanceoptimizationsneverstopJ

https://hortonworks.github.io/hdp-aws/s3-performance/index.html


Correctnesswithstrongconsistency

Ã Writeoperationsfollowedbyreadmaynotreturncorrect

results

– Issuesfordatapipelines,multi-stagejobs,etc.

Ã S3Guardproject:Intermediate,consistentmetadatastore

Ã WritecallsfromS3AFileSystemupdatebothS3andmetadata

store

Ã S3AFileSystemautomaticallytriestoreconcilemetadata

betweenS3andmetadatastoreonsubsequentreads

– Inconsistenciesarehandledbasedonpolicy

Ã Status:Inprogress

16

https://issues.apache.org/jira/browse/HADOOP-13345


SecuringdataaccessviaIAMRoles

Ã Integrationwithcloudprovider

Ã ProvideanIAMroleasinstanceprofileforacluster

Ã AttachpoliciesforaccessingS3totherole– E.g.Read-onlyaccessforBIclusterto

specificbuckets

Ã Status:Available


DataSecurityinHadoop

ApacheRangerÃ Finegrained,role-basedaccesspoliciesto

data– Table/columnlevelACL

Ã AuditaccessinformationÃ RowlevelfilteringÃ Dynamicdatamasking


DataGovernanceinHadoop

ApacheAtlasÃ Autodiscover&indexmetadataÃ TagdataÃ Trackdatalineage


Datagovernancetechnicalarchitecture– OnPremise

OnPremiseHDPCluster

RangerAdmin

PolicyPolicy

AtlasAdmin Metadata

GovernedHDPComponent(E.g.Hive)

RangerPlugin

AtlasPlugin

LDAP/AD

DataSteward


DataGovernanceintheCloud:Easeofadministrationwithflexibility

Ã Nolongerasinglecomputeclustergenerating/accessingdata

Ã Data&Metadataarestillsingleandshared

Ã EvolveAtlasandRangertobedatalakecentricthanclustercentric– SharedlongrunningAdmincomponents– Ephemeralpluginsoncomputeclusters

Ã Status:AvailableasaTechPreview

https://github.com/hortonworks/hdc-cli/blob/master/shared_cluster.md


SharedRanger/Atlasadminservices

AvailableinTechPreviewinHortonworksDataCloud

ETL-EDWCluster


LDAP/AD

RangerPlugin

AtlasPlugin

DataAnalyticsCluster


RangerPlugin

AtlasPlugin

RangerAdmin Policy

Policy

AtlasAdmin Metadata

CloudController

SharedEnterpriseServices

DataSteward


HDPCloudComputenodesonAWS

Ã RegularEC2instancesÃ CanattachEBSvolumesorephemeralstoragedisksÃ Groupedaccordingtofunctionality/accessrequirementsÃ Opportunisticprovisioning– spotinstances(workinprogress)

HDPCluster

MasterGroupGroup#1

Gatewaynode:Ambari

MasterGroupGroup#2

CloudController


HDPCloudComputenodesonAWS

24


Reliabilitywithcostbenefits

Ã HDPhostinstancescouldbecomeunhealthy– Unreliableunderlyinginfrastructure– Spotinstancesaretransient,dependentonbidprice– SLAimpactforworkloads

Ã Automaticallyreplaceun-healthynodes– Nocostsincurredifnodeisnotfunctional– Replaceunhealthyinstancestomaintainadesiredcapacity

Ã Status:Workinprogress


Auto-recoveryofslavenodes

Ã UseAmbaritodetectunhealthystatus&notifyCloudbreakÃ DecommissionandterminateunhealthyinstancesÃ Provisionnewinstancesandaddtocluster

HDPCluster

MasterGroupGroup#1

Gatewaynode:Ambari

MasterGroupGroup#2CloudController


Summary


OurConnectedDataPlatformSolutions

Hortonworks:PoweringtheFutureofData(Everybusinessisadatabusiness,mastervalueofdataviaopenapproach)

ModernDataApplications(CyberSecurity,IoT,Partners,Custom,etc.)

ConnectedDataPlatforms(ManageAllData:data-at-rest,data-in-motion,datacenter&cloud)

Training|Consulting|CommunityConnection|Partnerworks

DataCenterSolutions CloudSolutions

HortonworksDataCloudforAWS

AzureHDInsight

RackspaceAccentureOthers

HDP HDFSyncsortAtScale

PivotalHDBOthers

EnterpriseSubscription

SmartSense operationalsvc’s24x7SupportMaintenance

Etc.


http://hortonworks.com/info/aws-marketplace-credits-signup/


THANKYOU

introduction to hortonworks data cloud for aws

Software