trihug october: apache ranger

Hadoop Data Security with

Apache Ranger Biren Saini

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

© Hortonworks Inc. 2015

About me

•  Biren Saini •  Senior Solutions Engineer •  Governance SME Lead •  Overall 15 years of technology experience

@ Hortonworks }


Agenda

•  Hadoop Security Overview •  Apache Ranger

–  Introduction – Architecture – Sample Flow – Best Practices – Ranger Stacks – Demo


Overview of Security in Hadoop


5 Pillars of Security

•  Authentication •  Authorization

•  Audit

•  Encryption

•  Centralized Administration


Security Tools in Hadoop world

•  Kerberos (authentication) •  Apache Knox (authentication)

•  AD/LDAP (authentication)

•  Apache Ranger (authorization, audit, kms)

•  HDFS TDE (data encryption)

•  Wire Encryption (data protection)


HDFS

Typical Flow – SQL Access through Beeline client

HiveServer 2 A B C

Beeline Client


HDFS

Typical Flow – Authenticate through Kerberos

HiveServer 2 A B C

KDC

Login into Hive using AD password

Hive gets Namenode (NN) service ticket

Hive creates map reduce using NN ST

Client gets service ticket for Hive

Beeline Client

Active Directory


HDFS

Typical Flow – Add Authorization through Apache Ranger

HiveServer 2 A B C

KDC


Column level access control, auditing

Ranger

Beeline Client

File level access control

Active Directory

Import users/groups from LDAP

Login into Hive using AD password


HDFS

Typical Flow – Firewall, Route through Knox Gateway

HiveServer 2 A B C

KDC

Use Hive ST, submit query



Ranger

Knox gets service ticket for Hive

Knox runs as proxy user using Hive ST

Original request w/user id/password

Client gets query result

Beeline Client

Apache Knox

Active Directory


HDFS

Typical Flow – Add Wire and File Encryption

HiveServer 2 A B C

KDC

Use Hive ST, submit query



Ranger

Knox gets service ticket for Hive

Knox runs as proxy user using Hive ST

Original request w/user id/password

Client gets query result

SSL

Beeline Client

SSL SASL

SSL SSL

Apache Knox

Active Directory


Apache Ranger


Apache Ranger

•  Provides centralized policy definition for authorizing & auditing access to resources in a consistent manner.

•  Supported components as of v0.5 •  HDFS •  HBase •  Hive •  YARN •  Knox •  Storm •  Solr •  Kafka


Setup Authorization Policies

14

file level access control, flexible definition

Control permissions


Monitor through Auditing


Agent Agent Agent Agent Agent Agent

Apache Ranger authZ Architecture

HBase Hive YARN Knox Storm Solr Kafka

Agent

HDFS

Agent

Administration Portal (Ranger UI)

REST APIs

DB

SOLR

HDFS

Policy Server LDAP/AD user/group

sync

Log4j

KMS Audit Server

User Sync Server


Hadoop Cluster

Sample Simplified Workflow - HDFS

Policy Server

Audit Server

Administration Portal

Agent

Namenode

Audit Store

Ranger

Policy Store

Unauthorized user attempts to access the data

User access is denied

No Policy defined.


Hadoop Cluster


Policy Server

Audit Server


Admin sets policies for HDFS files/folder 1a

1b

1d

Agent

Namenode

Audit Store

Ranger

Policy Store

1c


Hadoop Cluster


Policy Server

Audit Server



Data scientist runs a map reduce job

User Application

Analysts access HDFS data through application

IT users access HDFS through CLI

1b

2a

2a

2a

Agent

Namenode

Namenode provides resource access to user/client

Namenode uses Agent for Authorization 2b

Audit Store

2d 2c

Ranger

Policy Store

1d

1c


Hadoop Cluster


Policy Server

Audit Server



Data scientist runs a map reduce job

User Application

Analysts access HDFS data through application

IT users access HDFS through CLI

1b

2a

2a

2a

Agent

Namenode

Namenode provides resource access to user/client

Namenode uses Agent for Authorization 2b

Audit Store

Admin requests the Audit report 3a 3b

3c

2d 2c

Ranger

Policy Store

1d

1c


Ranger UserSync Best Practice

21

•  Ensure LDAPS is used to integrate with Ranger •  Create OU ONLY for Hadoop users for performance •  Only run usersync when necessary

– How much users are being added and how often – How much users are changing roles – Too much syncing can degrade LDAP performance

•  Do not sync anonymously


Ranger Audit Best Practices

22

•  HDFS – Long term storage that can be used to understand user event

trends and predict anomaly •  RDBMS

– When SQL is preferred by auditors – MySQL, Oracle, Postgres, SQL Server

•  Solr – Nice quick reporting metrics to understand user event trends

•  Log4j Appenders


Ranger Stacks •  Apache Ranger v0.5 supports stack-model to enable easier onboarding

of new components, without requiring code changes in Apache Ranger.

Ranger Side Changes

Define Service-type

Secured Components Side Changes

Develop Ranger Authorization Plugin •  Create a JSON file with

following details : - Resources - Access types - Config to connect

•  Load the JSON into Ranger.

•  Include plugin library in the secure component. •  During initialization of the service: Init

RangerBasePlugIn & RangerDefaultAuditHandler class. •  To authorize access to a resource: Use

RangerAccessRequest.isAccessAllowed() •  To support resource lookup: Implement

RangerBaseService.lookupResource() & RangerBaseService.validateConfig()

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741207


Summary & Misc. points

24

•  All functions are available as Rest API •  Ranger integrates with AD/LDAP for ranger login as well as user sync. •  Support for High Availability (HA) •  Support for Transparent Data Encryption with KMS implementation •  Tighter integration with Apache Ambari •  Stack based implementation of Plugins •  Ranger also has the KMS for HDFS TDE. •  Some features in development are

–  Spark support –  Time based authorization –  Geo Location based authorization


Demo - HDFS

Admin

Sam Tom

/demo/data/trihug

/demo/data/trihug Ranger UI

WRITE Access denied READ Access denied

1

2

Sam Tom

/demo/data/trihug WRITE Access allowed READ Access allowed

3

Grants access

READ for Sam WRITE for Tom

Ranger Plugin

gets the update

WRITE Access denied

hdfs:hdfs rwx --- ---

Elevated Privileges Restricted Privileges

Directory already exists


Demo - Hive

Admin

Sam Tom

tickers eod

Ranger UI

WRITE Access denied READ Access denied

1

2

Sam Tom

WRITE Access allowed READ Access to SOME COLUMNS allowed 3

Grants access

READ for Sam ALL for Tom

Ranger Plugin

gets the update

WRITE Access denied

hive tables

tickers eod

hive tables

tickers eod

hive tables

SOME COLUMNS

READ Access to ALL COLUMNS denied

Created by “hive” user in trihug schema

Elevated Privileges Restricted Privileges

GRANT Access allowed

DB already exists


Demo time..


Thank you. Questions?

trihug october: apache ranger

Technology