Buzz MoschettiEnterprise Architect

buzzmoschettimongodbcombuzzmoschetti

BulletproofData Management

2

Part 3 In The Data Management Series

Validating Data

Software Best Practices

Safe Leverage

From RelationalTo MongoDB

ConqueringData Proliferation

BulletproofData Management

ccedilΩ

Part1

Part2

Part3

3

Congratulations At this Point Yoursquove

bull Created a Data Designbull Migrated Databull Built a PoC or maybe an Appbull Explored Operations

4

The Next Stage Defend amp Leverage

bull Document Validationbull Redactionbull Quality Of Service

5

MongoDB Doesnrsquot Have These Things

bull Document Validationbull Redactionbull Quality Of Service

YET^

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

2

Part 3 In The Data Management Series

Validating Data

Software Best Practices

Safe Leverage

From RelationalTo MongoDB

ConqueringData Proliferation

BulletproofData Management

ccedilΩ

Part1

Part2

Part3

3

Congratulations At this Point Yoursquove

bull Created a Data Designbull Migrated Databull Built a PoC or maybe an Appbull Explored Operations

4

The Next Stage Defend amp Leverage

bull Document Validationbull Redactionbull Quality Of Service

5

MongoDB Doesnrsquot Have These Things

bull Document Validationbull Redactionbull Quality Of Service

YET^

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

3

Congratulations At this Point Yoursquove

bull Created a Data Designbull Migrated Databull Built a PoC or maybe an Appbull Explored Operations

4

The Next Stage Defend amp Leverage

bull Document Validationbull Redactionbull Quality Of Service

5

MongoDB Doesnrsquot Have These Things

bull Document Validationbull Redactionbull Quality Of Service

YET^

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

4

The Next Stage Defend amp Leverage

bull Document Validationbull Redactionbull Quality Of Service

5

MongoDB Doesnrsquot Have These Things

bull Document Validationbull Redactionbull Quality Of Service

YET^

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

5

MongoDB Doesnrsquot Have These Things

bull Document Validationbull Redactionbull Quality Of Service

YET^

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

6

What Can We Do Today

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

Write Some Code

1 Focus on interfaces2 Design for change3 Keep application data access layer

data management logic and database io well-factored

4 Minimize compile-time binding

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

8

Starting Point The Data Access Layer

MongoDBJava Driver

Data Access Layer

Application

class DataAccessLayer private String authenicatedID private String effectiveID private Role role

init() MongoClient mc = new MongoClient (args) DB db = mcgetDB(args)

List getTransactions(Map predicate) Map mql = doWhateverYouNeed(predicate)

DBCollection coll = dbget(ldquoTXrdquo) DBCursor c = collfind(mql)

while(chasNext()) Map raw = (Map) cgetNext() Map morphed = myMorphingLogic(raw) listadd(morphed) return list

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

9

Document Validation

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

10

A Query Filters Outbound Data

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

11

How About Using It To Filter Inbounds

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

12

$exists And $type Already in MQL

ldquonamerdquo$type2

$or[ldquoagerdquo$existsfalse ldquoagerdquo$type16 ]

$and [ $name $type2 $or[ $and[weightrdquo$type16 height$type16] $and[weightrdquo$exists0 height$exists0] ]])

Ensure ldquonamerdquo exists (because not null) and is a string

ldquoagerdquo optional but if exists must be a 32bit integer

ldquonamerdquo required as string and weight and height both required integers or both not present

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

13

hellip And MQL Goes Way Beyondhellip

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

14

A New MQL Validator Module Emerges

class MQLValidator ValidationResult validate(Map MQL Map data)

MongoDBJava Driver

Data Access Layer

Application

Validator NOT inline to MongoDB driverbull Interface too big to create a faccediladebull Beware of ldquotall stacksrdquo

MQLValidator

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

15

MongoDBDB Engine

Migrating Capability into MongoDB

MongoDBJava Driver

MQLValidatorJava

Data Access Layer

MongoDBDB Engine

MongoDBJava Driver

MQLValidator

JavaData Access

Layer

bull Coming in v32bull Investment in validation design preservedbull Validation enforceable through ALL drivers

and languages

MongoDBPython Driver

Application Application

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

16

Code For The FuturehellipToday

class DataAccessLayer

someWriteOperation(Map data)

if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else Map mql = getMQL() wersquoll see this shortly $or[ldquoagerdquo$existsfalse ldquoagerdquo$type16] ValidationResult vr = MQLValidatorvalidate(mqldata) if(vrok()) collectioninsert(data)

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

17

But What About Today

MQLValidator

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

18

Temporary Filling PQL

bull (P)refix (Q)uery (L)anguagebull Database independent filter of Mapsbull Similar to MQLbull 450 lines of Javabull moschettiorgrantsPQLhtml

PQL

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

19

Bridge MQL to PQL

class MQLValidator private PQLFilter pqlfilter

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else Map pqlfilter = convertMQLtoPQL(mql) or[ldquonullrdquo ldquoagerdquo ldquotyperdquo ldquoagerdquo ldquoINTrdquo] rc = pqlfiltereval(pql data) return rc Map convertMQLtoPQL(Map mql) ~200 lines

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

20

No PQL No Problem

class MQLValidator

validate(Map mql Map data) boolean rc if(MQLValidationAvailableAsLibrary) rc = ActualMongoDBMQLvalidate(mql data) else SomeType yo = convertMQLtoYourThing(mql) rc = YourFilter(yo data) return rc SomeType convertMQLtoYourThing(Map mql)

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

21

MQL Is Easy To Navigate

$or[ $and[ ldquonamerdquo $type2 ldquonumCluesrdquo $gt 0 ldquonumCluesrdquo$type16 ldquobirthdayrdquo $type 9 ldquohiredaterdquo $type 9 $or [ldquoprefsrdquo$existsfalse ldquoprefsrdquo$type3 ] ] ldquonamerdquo $existsfalse ]

bull ldquoWalkrdquo not ldquoparserdquobull Operators distinct from operandsbull Operands are native type (eg Date)

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

22

Where Do Validations Come From

The Database

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

23

The Validations Collection

gt dbvalidationsfind() ldquocollectionNamerdquo ldquoproductrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo __$or[ldquoagerdquo__$existsfalse ldquoagerdquo__$type16 ] ] ldquocollectionNamerdquo ldquotransactionrdquo ldquovalidationsrdquo [ ldquonamerdquo ldquofrontOfficerdquo ldquotyperdquo ldquoMQLrdquo ldquoexprrdquo hellip lots of MQL here hellip ]

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

24

Various ldquoLevelsrdquo of Validation

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquodefaultValidationrdquo ldquoinitialSetuprdquo ldquovalidationsrdquo [ ldquonamerdquo ldquoinitialSetuprdquo hellip ldquonamerdquo ldquofrontOfficerdquo hellip ldquonamerdquo ldquomiddleOfficerdquo hellip ldquonamerdquo ldquobackOfficerdquo hellip ]

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

25

Multiple Types Schema By Example

gt dbvalidationsfind() ldquocollectionNamerdquo ldquofoordquo ldquovalidationsrdquo [ ldquonamerdquo ldquosimplerdquo ldquotyperdquo ldquoSBErdquo ldquoexprrdquo ldquonamerdquo ldquostringrdquo ldquoagerdquo ldquointegerrdquo ldquopetNamesrdquo [ ldquostringrdquo ] ldquobdayrdquo ldquodaterdquo ]

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

26

The Stack So Far

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

ValidatorDBUtils populates an MQLValidator object from MongoDB

PQLFilter

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

27

Representative Example

class DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) dbvalidations

someWriteOperation(Map data) if(ValidationEnabledInMongoDBengine) collectioninsert(data) Not yet else String vn = ldquoappropriateValidationRulesNamerdquo ValidationResult vr = vvvalidate(collname vn data)) if(vrok()) collectioninsert(data)

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

28

Redaction

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

29

Concept Post Query Operations (PQO)

ssn $hash model birthdate null

$and[ldquonamerdquordquobuzzrdquoldquoprefsrdquo$existstrue]

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

30

Adopt MQL-like behavior

ldquossnrdquonull

ldquoaddressrdquo ldquoXXXXrdquo

ldquossnrdquo $substitute ldquossnmodelrdquo

Remove field by setting to null

Redact address with fixed value

Substitute SSN with a different correct consistent value

ldquocounterpartyrdquo $hash ldquoMD5rdquo

Hash counterparty name to consistent value

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

31

A New PostQuery Module Emerges

class PostQuery process(Map data Map operations)

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

32

Where Do PQOs Come From

The Database

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

33

The Postquery Collection

gt dbpostqueryfind() ldquocollectionNamerdquo ldquoproductrdquo ldquooperationsrdquo [ ldquonamerdquo ldquobasicPIrdquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo ldquossnrdquonull ] ldquocollectionNamerdquo ldquocustomerIndordquo ldquooperationsrdquo [ ldquonamerdquo ldquopersonalDatardquo ldquotyperdquo ldquoPQOrdquo ldquoexprrdquo hellip lots of PQO here hellip ]

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

34

The Stack Is Getting Rich

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

35

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db)

someWriteOperation(Map data) hellip

someReadOperation(Map pred) Map mql = convertToMQL(pred) Map data = collectionfind(mql) String pqon = mapRoleToRulesName() ppprocess(collname pqon data) in place update return data

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

36

Quality of Service

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

37

QOS In Action

class DataAccessLayer someReadOperation(Map pred) if(qosblackout(ldquosomeReadOperationrdquo)) throw QOSOperationDenied

Map mql = convertToMQL(pred) int ms = qosgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxTime(msTimeUnitMILLISECONDS) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

38

Where Do We Store QOS Values

The Database

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

39

The QOS Collection

gt dbqosfind() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquostdrdquo ldquomaxtimerdquo 250

ldquofunctionrdquo ldquosomeReadOperationrdquo ldquorulerdquo ldquoreportingrdquo ldquoblackoutrdquo ldquostartrdquo ldquo0800rdquo ldquoendrdquo ldquo1700rdquo ldquomaxtimerdquo 2000 ldquo hellip rdquo ]

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

40

QOSDBUtils

Coming Togetherhellip

PostQuery

MongoDBJava Driver

MQLValidator

Data Access Layer

Application

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

41

Representative Exampleclass DataAccessLayer MQLValidator vv = new MQLValidator() NOT DB dependent PostQuery pp = new PostQuery() QOS qs = new QOS()

init() DB db = mongoClientgetDB( rdquomydb ) ValidatorDBUtilspopulate(vv db) PQODBUtilspopulate(pp db) QOSDBUtilspopulate(qs db)

someReadOperation(Map pred) Map mql = convertToMQL(pred) String role = getRole() somehow int maxms = qsgetMaxTime(ldquosomeReadOperationrdquo role) Map data = collectionfind(mql)maxtTime(maxms tu) String pqon = ldquoappropriatePQORulesNamerdquo ppprocess(collname pqon data) in place update return data

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

42

QOSDBUtils

A Highly Leveragable Investment

PostQuery

MQLValidator

Data Access Layer 1

Application1

ValidatorDBUtils

PQLFilter

PQODBUtils

QOS

Application2

Data Access Layer 2

Application3

Application4

Data Access Layer 3

Application5

Application6Reusable For ALL Data Access Layer Logic

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

43

Not Just Java Not A Problem

DAL operations have little or no statehellip

Data and MQL and diagnostics easily and losslessly converted to and from JSONhellip

Can you say hellip Web Service

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

44

A Really Nice Stack

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorDBUtils

PQLFilter

HTTP Endpoint

pythonApplication

curl

JSON over HTTP(S)

JSONlt-gtJava Maps

QOSDBUtils

PostQuery

PQODBUtils

QOS

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

45

What Can We Do

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

46

Secure Access To Redacted Data for Testing

$ curl ndasho contactsjson -H X-Portal-IdtestID -H X-Portal-PWthePassword httpsrefdata8080customersop=findamppredicate=lsquoldquonamelastrdquo ldquoJonesrdquorsquo

$ head -1 contactsjson ldquonamerdquo ldquofirstrdquordquoBobrdquo ldquolastrdquordquoJonesrdquo ldquolocationrdquordquoNA-EASTrdquo ldquossnrdquo ldquo000-00-0000rdquo ldquohiredaterdquo ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquo

$ mongoimport ndash-host testHost ndashd testdb ndashc contacts contactsjson15 items inserted

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

47

Get It Programmatically Too

This JSON parser observes MongoDB type metadata conventions eg ldquo$daterdquo ldquo2015-04-22T170454580-0400rdquoimport commongodbutilJSON

getData() String url = httpsrefdata8080customershelliprdquo URLConnection con = new URL(url)openConnection() InputStream response = congetInputStream() BufferedReader in = new BufferedReader(response)

String doc while((doc = inreadLine()) = null) Map data = JSONparse(doc) datassn = ldquo000-00-0000rdquo datehiredate = javautilDate 2015-04 hellip datanamefirst = ldquoBobrdquo

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

48

Robust Validated Data Ingest

$ curl ndashd tradesjson -H X-Portal-Idprodadm -H X-Portal-PWthePassword httpsrefdata8080tradesop=load -o responsejson

$ cat responsejson ldquoassignedBatchIDrdquo ldquoB123rdquo ldquonumItemsExaminedrdquo 13245 ldquonumItemsInsertedrdquo 13242 ldquonumItemsRejectedrdquo 3 ldquoerrorsrdquo [ type ldquovalfailrdquo rule ldquofront hellip ] ldquobatchMD5rdquo ldquoe19c1283c925b3206685ff522acfe3e6rdquo

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

49

Concept The control_ Collection

gt show collectionsbookscontrol_customerfirmsgt dbcontrol_find() ldquocollectionNamerdquo ldquoproductrdquo ldquoqosrdquo [ hellip ] ldquovalidationsrdquo [ hellip ] ldquooperationsrdquo [hellip ]

bull Single namespace for capabilitiesbull Easier to add new capabilitiesbull Tighter (therefore better) securityentitlement

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

50

Validation QOS and PQO via Web Services

MongoDBJava Driver

MQLValidator

Data Access Layer

JavaApplication

ValidatorHTTPUtils

PQLFilter

pythonApplication

curl

JSON over HTTP(S)

QOSHTTPUtils

PostQuery

PQOHTTPUtils

QOS

HTTP Service

JSONlt-gtJava Maps

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

51

Are We Excited Yet

Contact me or MongoDB forbull Beta program for 32 featuresbull Access to MQLValidator PQO

and other Java resources

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

Questions amp Answers

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

54

Concept DataProvider

public interface DataProvider init() fetch(String collection Map mql) insert(String collection Map data) update(String collection Map mql Map newData)

Class MongoProvider implements DataProvider hellip

Class RESTfulProvider implements DataProvider hellip

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())

55

The RESTful Provider

class RESTfulProvider implements DataProvider init() setup HTTP machineport endpoint

fetch(String collection Map mql) String jsonstr = JSONUtilstoJSON(mql) String url = construct(collection jsonstr)

url is httpmachineportcollectionNameop=findampmql=lsquoldquoproductrdquordquocleanserrdquordquoexpiresrdquo $gt $date ldquo20200101rdquorsquo

HTTPResponse res = call(url) Map data = JSONUtilsfromJSON(resgetContent())