working with strongly typed data models in lua for … with strongly typed data models in lua for...

WorkingwithstronglytypeddatamodelsinLuaforbuildingIndustrialInternetofThings(IIoT)applica=ons

TheConnec=vityPlaAormfortheIndustrialInternetofThings™

RajiveJoshi,Ph.D.PrincipalSolu7onArchitect,Real-TimeInnova7ons,Inc.

LuaWorkshop2016SanFrancisco,CA

2016Oct14

Agenda

•  Background•  Problem•  Solu=on•  Tutorial|Demo•  Applica=ons•  NextSteps

©2015Real-TimeInnova=ons,Inc.

Background

Whatwedo?

TheIndustrialInternetofThingsIndustrialInternetofThings(IIoT)ConsumerInternetofThings(CIoT)

Cyber-PhysicalSystems(CPS)

RTIExcelsatIndustrialIoT(IIoT)•  ~1000Projects

–  Healthcare–  Transporta=on–  Communica=ons–  Energy–  Industrial–  Defense

•  15+Standards&Consor=aEfforts–  Interoperability–  Mul=-vendorecosystems

ConnectAnything,Anywhere-Intelligently!

•  Proximity•  PlaAorm•  Language

•  Physicalnetwork•  Transportprotocol•  Networktopology

RTIConnextDDS“DataBus”

Seamlessdatasharingregardlessof:

ü Reliableü Real-Timeü Scalableü Availableü Resilientü Secureü Safeü Composable

RTIConnextPlaAormisbuiltontheDDSStandard


hdp://portals.omg.org/dds/

ProblemDefini=on

“Datadominates.Ifyou'vechosentherightdatastructuresandorganizedthingswell,thealgorithmswillalmostalwaysbeself-evident.Datastructures,notalgorithms,arecentraltoprogramming.(SeeBrooksp.102.)”

-  RobertPike

Data-CentricSystems:SharedDataSpace


ComponentX1

Data-Objects(Topic) Component

Y1DataWriter

write()

DataReader

take()read()



ComponentX1


Y1

ComponentXN

ComponentYM

DataWriter

write()

DataReader

take()read()



ComponentX1


Y1

ComponentXN

ComponentYM

DataWriter

write()

DataReader

take()read()

C/C++ Java

.Net/C# Lua

Pickthelanguagemostsuitabletotherequirementsofacomponent

Mul=pleplaAorms,Mul=pleLanguages

Data-CentricSystems:DataModels


DataModel(Datatype)

structShapeType{longx;longy;doubleshapesize;string<128>color;//@Key};

ComponentX1

ComponentY1

ComponentXN

ComponentYM

C/C++ Java

.Net/C# Lua



DataModel(Datatype)


ComponentX1

ComponentY1

ComponentXN

ComponentYM

C/C++ Java

.Net/C# Lua

InterfaceDefini7onLanguage(IDL)XMLwithEquivalentSchema

DDSX-TypesSpecifica=on



DataModel(Datatype)


ComponentX1

ComponentY1

ComponentXN

ComponentYM

C/C++ Java

.Net/C# Lua

struct { : }

public class{ : }

public class { : }

InterfaceDefini7onLanguage(IDL)XMLwithEquivalentSchema

DDSX-TypesSpecifica=on

?

DataModelConstrainstheData-Objects


DataModel(Datatype)


Data-Objects(Topic)

Constrains

LanguageNeutral LanguageSpecificSta7c:C/C++,Java..NET/C#,Ada.…Dynamic:Lua,Python,Javascript,…

Compilerenforcesinasta5callytypedlanguage

DataModelConstrainstheData-Objects


DataModel(Datatype)


Data-Objects(Topic)

Constrains

LanguageNeutral LanguageSpecificSta7c:C/C++,Java..NET/C#,Ada.…Dynamic:Lua,Python,Javascript,…

Howtoenforceindynamicallytypedlanguages?HowtoenforceinLua?

Solu=on

DDSL:DataDomainSpecificLanguage

DDSL:DataDomainSpecificLanguageAnalgebrafordefining,using,andmanipula5ngX-Types(inLua)


DataModel(Datatype)


Data-Objects(Topic)

template(Lua)

instances(Lua)

DDSL

EnforcesConstraintsKeepsSynchronized

Create&Modify•  Datatypes•  InstancesIndexdatafields

Datamodelscanbeverycomplex….Primi7ve

•  Atomic(long,double,string,etc…)

•  Const•  Enums

Structural•  Structs•  Unions•  Sequences(mul=dimensional)•  Arrays(mul=dimensional)


Organiza7onal•  Typedefs•  Module•  Annota=ons

WithoutDDSL…

• Thedatatypemustbemaintainedoutsidethescrip=nglanguageenvironment– notaccessibletoprogram(inthelanguage)

• Thedatatypemustbeknowntotheprogrammer,anditsstructurehard-codedinthescript– error-proneduetoduplica=on– bridletodatatypeevolu=onorlarge/complextypes

• Ascriptcannotuseorintroducenewtopicsorendpoints(notknowntotheprogrammer)– impossibletobuildgenericcomponents


WithDDSL…

• Thedatatypecanbemaintainedinsidethescrip=ngenvironment– fullyaccessibletoprogram

• Thedatatypeneednotbeknowntotheprogrammer,anditsstructureneednotbehard-codedinthescript– safe,singleversionoftruth(nodatatypeduplica=on)– robusttodatatypeevolu=onorlarge/complextypes

• Ascriptcanuseorintroducenewtopicsorendpoints(notknowntotheprogrammer)– possibletobuildgenericcomponents


QuickExample…


DataModel-IDLstructShapeType{longx;longy;doubleshapesize;string<128>color;//@Key};

Data-Object-Luashape={x=50,y=30,shapesize=20,color=‘GREEN’}

QuickExample…

DataModel–DDSL(template)

localShapeType=xtypes.struct{ShapeType={{x={xtypes.long}},{y={xtypes.long}},{shapesize={xtypes.long}},{color={xtypes.string(128),xtypes.Key}},}}

Data-Object–DDSL(instance)

localshape=xtypes.new_instance(ShapeType)shape.x=50shape.y=30shape.shapesize=20shape.color='GREEN'


DataModel-IDLstructShapeType{longx;longy;doubleshapesize;string<128>color;//@Key};

Data-Object-Luashape={x=50,y=30,shapesize=20,color=‘GREEN’}

DDSLopensupnewpossibili=es...•  ADDSLdatatype

– Canbeexaminedandtraversedintheprogram– Canbedefinedandmanipulatedbytheprogram•  create|read|update|modifyopera=ons

– Canbesynthesizedandmanipulatedbytheprogram– Canbeusedtoautomatecodeordatagenera=on

whileusingthefullpowerandflexibilityofLua•  BringsdatatypestoeveryIoTplaAorm

– DDSLisplaAormindependent•  iswrideninpureLuawithnoexternaldependencies(notevenDDS!)•  canbeusedonanyplaAormonwhichLuacanbeused(prac=callyeverywhere)

– Evenweb-browsers(e.g.usinglua.vm.js)©2015Real-TimeInnova=ons,Inc.

DDSL

Analgebrafordefining,using,andmanipula=ngX-Types(inLua)

So,whatexactlyisDDSL?

1.  ReplacementforIDL,byallowingdatatypesdefini=onsinLua2.  Factoryofdata-objects(instances)thatadheretoadatatype3.  Analgebratooperateondatatypesandmaintainthe

associateddata-objects(instances)4.  Generatorofaccessorstringsforindexingmembersinadata-

objectstore5.  Anextensibleframeworkfordatamodeling


1.IDLReplacementinLua

• OMGDDSX-TypesSpecifica=oninLua– includingfeaturesnotyetsupportedinIDL• e.g.customannota=ons

• XML→DDSL– ImportdatatypesdefinedinXML

• DDSL→IDL(→XML)– ExportdatatypestoIDL

• Easytodefinenewkindsofdatatypes– Usefulforexperimen=ngwithnewX-Typesfeatures• e.g.choice

1.IDLReplacementinLua

localMAX_COLOR_LEN=xtypes.const{MAX_COLOR_LEN={xtypes.long,128}}localShapeType=xtypes.struct{ShapeType={{x={xtypes.long}},{y={xtypes.long}},{shapesize={xtypes.long}},{color={xtypes.string(MAX_COLOR_LEN),xtypes.Key}},}}


localMAX_COLOR_LEN=xtypes.const{MAX_COLOR_LEN={xtypes.long,128}}localShapeType=xtypes.struct{ShapeType=xtypes.EMPTY}ShapeType[1]={x={xtypes.long}}ShapeType[2]={y={xtypes.long}}ShapeType[3]={shapesize={xtypes.long}}ShapeType[4]={color={xtypes.string(MAX_COLOR_LEN),xtypes.Key}}

Declara=vestyle(DatatypeConstructor)

Impera=veStyle(DatatypeCRUD)

2.Factoryofinstances

• Instancesareconstrainedbythedatatype– arbitrarymemberscannotbeaddedtoinstances– membersmaybeassignednil(e.g.op=onal);andlaterre-assignedanon-nilvalue– collec=on(arrayandsequence)boundsareenforced;nowallowedtoexceedthemaxcapacity

• Yet,instanceperformanceisonparwithamanuallydefinedinstance– nooverheadinsewng/gewngmembervalues

•  itwasanexplicitdesigngoaltonotaddanyoverheadoverrawmemberfieldseders/geders

– onlystructuralconstraintsareenforced!•  type-checkingisnotenforcedformembervalueassignment(althoughanapplica=oncouldeasilydosoifdesired)


2.Factoryofinstances--shapeisequivalenttomanuallydefiningthefollowing--localshape_manual={x=50,y=30,shapesize=20,color='GREEN',}


localshape=xtypes.new_instance(ShapeType)print("---Iteratethroughinstancemembers:unordered---")forrole,_inpairs(shape_manual)doshape[role]=shape_manual[role]print('\t',role,shape[role])end

--- Iterate through instance members : unordered --- color GREEN x 50 shapesize 20 y 30

⇒

3.DatatypeAlgebra

• Datatypescanbemutated(unlikeIDL/XML)– the“kind”(meta-datatype)isimmutableanddefinesthestructureandconstraints– allotheraspectsofadatatypecanbechanged,e.g.:name,members,annota=ons,inheritance,switch

•  Datatypechangesarepropagatedtoinstances(unlikeIDL/XML)– addingamembertothedatatypealsoaddsittoalltheinstances(withadefaultvalue)– removingamemberfromathedatatypealsoremovesitfromalltheinstances– changingamember'sdatatypepropagatesthechangetoalltheinstances


3.DatatypeAlgebra

ModelsandInstancesADDSLinstancehastwofaces(likeacoin):•  anunderlyingdatatypemodel(theblueprint)•  fieldswithrolenamesdictatedbythedatatype

Constructorsreturnaspecial“template”instance(e.g.MyType)asahandletothe

underlyingmodel.

Datatypemodelsaremanipulatedusingoperatorsoninstances.Anyoftheinstancescouldbeused,butitsidioma=ctousethetemplateinstance.Templateinstanceslinkmodelelementstoformadatamodelgraph.


MyTypemember_role_1 str|tbl

:

member_role_M str|tbl

MyTypeModelOperatorsi-thmember MyType[i]={role={…}}

#members #MyType

name MyType[NAME]

namespace MyType[NS]

kind MyType[KIND]

() capacity|constvalue|alias

qualifiers|base|switch

MyType[QUALIFIER|BASE|SWITCH]

new_instance()myInstance

member_role_1 value|tbl

:

member_role_M value|tbl

metatable

template

3.DatatypeAlgebra

ModelsandInstancesADDSLinstancehastwofaces(likeacoin):•  anunderlyingdatatypemodel(theblueprint)•  fieldswithrolenamesdictatedbythedatatype

Constructorsreturnaspecial“template”instance(e.g.MyType)asahandletothe

underlyingmodel.

Datatypemodelsaremanipulatedusingoperatorsoninstances.Anyoftheinstancescouldbeused,butitsidioma=ctousethetemplateinstance.Templateinstanceslinkmodelelementstoformadatamodelgraph.



:


MyTypeModelOperatorsi-thmember MyType[i]={role={…}}

#members #MyType

name MyType[NAME]


kind MyType[KIND]






:


metatable

metatable

template

3.DatatypeAlgebra:ModelOperatorsprint('---addmemberz---')ShapeType[#ShapeType+1]={z={xtypes.string(),xtypes.Key}}print_datatype(ShapeType,'ShapeType:')print('---removememberx---')ShapeType[1]=nilprint_datatype(ShapeType,'ShapeType:')print('---redefinemembery---')ShapeType[1]={y={xtypes.double}}print_datatype(ShapeType,'ShapeType:')print('---addabasestruct---')localProperty=xtypes.struct{Property={{name={xtypes.string(MAX_COLOR_LEN)}},{value={xtypes.string(MAX_COLOR_LEN)}},}}ShapeType[xtypes.BASE]=Propertyprint_datatype(ShapeType,'PropertyShapeType:')print_instance(ShapeType,'PropertyShapeTypeaccessors:')


4.GeneratorofAccessorStrings•  Datatypeisaspecial‘template’instance

– whosemembersareini=alizedtotheaccessorstringsforaDDSDynamicDataobject– automa=callymaintainedtoconfirmwiththeunderlyingdatamodel– forefficiency,collec=onmemberelementsareallocated(atmostonce)ondemand(i.e.whenaccessed)


--ShapeTypecanbeaccessedasifitwasdefinedas--ShapeType={x=‘x’,y=‘y’,shapesize=‘shapesize’,color=‘color’}

4.Anextensibleframeworkfordatamodeling

•  Coreenginesupportsagenericmeta-model–  Agnos=ctospecificdatamodelinglanguageconstrains–  X-TypesmoduleextendstheddslenginetosupportDDSX-Types(OMGIDL)

•  Easytoextendthemodelinglanguage– Newatomicdatatypes,annota=ons,composi=ons,…

•  Easytoaddnewconstraints– Dataranges– Dataunits–  Seman=cs


DatatypeAlgebra

DDSLNomenclature



:


MyTypeAPIi-thmember MyType[i]={role={…}}

#members #MyType

name MyType[NAME]


kind MyType[KIND]






:


metatable

metatable

template

NomenclatureGeneralConcept DDSLTerminology

instance(a.k.a.data-object) instance

datatype template

member

membername role

memberdefini=on roledefini=on(roledef)

annota=on qualifier

DatatypeAlgebra-ModelConstructors


Createadatatypetemplate:

MyType = xtypes.<kind>{...}

where,<kind> is one of: annotation,

const, enum, struct, union, module, typedef

and{...} dependson<kind> e.g. MyType = xtypes.struct{MyType=xtypes.EMPTY} Commondefini=onstyle:

{ <name> = { <definition> } }

DatatypeAlgebra-ModelOperators


Createamember: MyType[i] = { role = {template, [array|sequence,] [[annotation,]...] } Deleteamember: MyType[i] = nil Namespace,Qualifiers,Base(structs),Switch(unions): MyType[xtypes.NS] = AnotherType MyType[xtypes.QUALIFIERS] = { annotation, ... } MyType[xtypes.BASE] = BaseType MyType[xtypes.SWITCH] = DiscriminatorType Name: MyType[xtypes.NAME] = ‘NewTypeName’ tostring(MyType) Kind(immutable): MyType[xtypes.KIND]

DatatypeAlgebra-ModelIterators


structsIterateoverthemembers:

for i = 1, #MyType do local role, roledef = next(MyType[i]) print(role, table.unpack(roledef)) end

unionsIterateoverthemembers:

for i = 1, #MyType do local case = MyType[i] -- case discriminators: print(table.unpack(case)) -- member: local role, roledef = next(case, #case) print(role, table.unpack(roledef)) end

DatatypeAlgebra-InstanceConstructors


Createaninstance: myInstance = xtypes.new_instance(MyType)

Createacollec=onofinstances: myCollection = xtypes.new_collection(MyType,

capacity) NOTE:Asequenceorarraymemberinaninstanceisacollec=on

DatatypeAlgebra-InstanceCollec7ons


Isitacollec=on?if xtypes.is_collection(myCollection)then

print( ‘capacity’, myCollection() ) print( ‘length’, #myCollection )

end Iterateoveracollec=on’slength: for i = 1, #myCollection do print( myCollection[i] ) end Iterateoveracollec=on’scapacity: for i = 1, myCollection() do print( myCollection[i] ) end

DatatypeAlgebra-InstanceIterators


Iterateoverinstancemembers(unordered): for role, value in pairs(myInstance) do print(role, value) end Iterateoverinstancemembers(ordered): for i = 1, #MyType do local element_i, role = MyType[i] role = next(element_i) -- struct

role = next(element_i, #element_i)-- union local value = myInstance[role] print(role, value, table.unpack(MyType(role))) end

DatatypeAlgebra–TemplateInstance


IterateoverDDSDynamicDataaccessorstrings(unordered): for role, accessor in pairs(MyType) do print(role, accessor) end IterateoverDDSDynamicDataaccessorstrings(ordered): for i = 1, #MyType do local element_i, role = MyType[i] role = next(element_i) -- struct

role = next(element_i, #element_i)-- union local accessor = MyType[role] print(role, accessor) end

Fieldvaluesareaccessorstringsforindexingmembersinadata-objectstore

DatatypeAlgebra–MiscOperators


Getthefullyqualifiednamewithinthenamespaces print( xtypes.nsname(MyType) ) Resolveanalias(e.g.typedef)toitsunderlyingdatatype print( xtypes.resolve(MyType) ) Getaninstance’sunderlyingdatatypetemplate print( xtypes.template(myInstance) )

DatatypeAlgebra–U7li7es

• ddsl.xtypes.u=ls– to_instance_string_table(instance)– to_idl_string_table(instance)– nslookup(name,ns)

• ddsl.xtypes.xml– filelist2xtypes(filename_array)– file2xtypes(filename)– string2xtypes(xmlstring)

• Commandline– runxml2ddsl[-d]<xml-file>[<xml-files>...]


Tutorial

Runningthetutorial… git clone github.com/rticommunity/rticonnextdds-ddsl/

export DDSL_HOME=$(pwd -P)

export LUA_PATH+="${DDSL_HOME}tutorial/?.lua;”

cd tutorial/

lua ddsl_tutorial.lua

Applica=ons


CurrentApplica=ons(thatIamawareof)

• DataEmulator– RTIExperimentalProduct•  Emulatedata-centricinterfaces•  Firstdroptocustomer(2015Jun30)

• DataGeneratorsforReac=veProgramming– ContribuedbySumantTambe– Usefulforsimula=onandtes=ng

Poten=alApplica=ons•  Communityofopendatamodelsfor[Industrial]IoT•  Dataserializa=onanddeserializa=on•  Codegenera=on

–  Integra=onwithtempla=nglibraries• Modelmappingandgenera=on– synthesizedatatypesfromseman7cmodels– synthesizedata-modelmappings

• Synthesisofdatamodels– machinelearning

• Experimenta=onwithNewDataModelingIdeas


NextSteps

•  GewngStartedGuide– README

•  hdps://github.com/r=community/r=connextdds-ddsl

•  UsersManual– API

•  hdp://r=community.github.io/r=connextdds-ddsl/• UnitTests– test/(formoreadvancedexamples)• ddsl-xtypes-tester.lua


Apache2.0

ThankYou!

rajive@r=.com|@RajiveJoshi

WeareHiring!www.r7.com/company/careers.html

working with strongly typed data models in lua for … with strongly typed data models in lua for...

Documents