LearningandInferenceinDynamicEnvironments

KatsumiInoue�PrinciplesofInforma8csResearchDivisionNa8onalIns8tuteofInforma8cs,Japan

�  InoueLabmembers:KatsumiInoue,NicolasSchwind,MorganMagnin*1,TendaOkimoto*2,TonyRibeiro*1,MaximeClement,Kotaro

Okazaki,TaisukeSato*3

(*1:ÉcoleCentraledeNantes,*2:KobeUniversity,*3:AICenter,AIST)

NIIIABMee(ng,October28,2015

SNS

Disaster

Security

Transportation

Unexpected Events

Economics

Big,Changing,Con8nuous,

Real-Time,andIncompleteData

DataMining(Sta8c)

Real-TimeProcessing

（NoInference）

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

Aninternalmodelisconstructedbylearningfromenvironmentandinterac8onwithothersystems.Themodelisusedforchoosingthenextac8on.

Theeffect/resultofanac8onaffectstheenvironmentandupdates8me-seriesdata,history,experienceandgoals.Thenthemodelisupdatedaccordingly.

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰Ac@on

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?Theagentalsointeractswithotheragents,anditsinternalmodelisrefinedbysuchinterac8ons． Ac@on

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰Ac@on

Ac@onDynamicTransi8onGoal:DevelopAItechniquesthatcaniden8fythemodelofasystemindynamicenvironmentsandcanachievetasksevenwhenunknown

situa8onsareencountered.

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰Ac@on

Ac@onMachineLearning

DynamicTransi8on

TheMachineLearningPartconstructsthemodeloftheagentbyabduc8onandinduc8onfromthegoal,data,historyandbackgroundknowledge.

DecisionMaking

LearningFromInterpreta8onTransi8on(LFIT)

LearningDynamicalandComplexNetworks•  Learningdynamicsystemsinvolvingposi8veandnega8vefeedbacks•  LearningBooleannetworksfromstatetransi8ondiagrams•  LearningCellularAutomatafromtracesofconfigura8onchange

Ø  Inoue,K.,Ribeiro,T.,Sakama,C.:“LearningfromInterpreta8onTransi8on”,MachineLearning,94(1):51-79,2014.

Ø  Völker,M.,Inoue,K.:“LogicProgrammingforCellularAutomata”,ICLP2015.

currentpa\ern 111 110 101 100 011 010 001 000newstateforcentercell 0 1 1 0 1 1 1 0

•  c(x,t+1)←c(x-1,t)∧c(x,t)∧￢c(x+1,t).•  c(x,t+1)←c(x-1,t)∧￢c(x,t)∧c(x+1,t).•  c(x,t+1)←￢c(x-1,t)∧c(x,t)∧c(x+1,t).•  c(x,t+1)←￢c(x-1,t)∧c(x,t)∧￢c(x+1,t).•  c(x,t+1)←￢c(x-1,t)∧￢c(x,t)∧c(x+1,t).

t 0 1 2 3 4

0

1

2

3

4

5

6

7

8

9

Wolfram’sRule110(Turing-complete)

LearningRobustBooleanNetworks

Li,F.etal.:Theyeastcell-cyclenetworkisrobustlydesigned,PNAS,101(14),2004.

• Mosttransi8onsfrom212statesbelongtothesamebasinofa2racKon.

•  Fromthisstatetransi8on,LFITlearned54statetransi8onrulesin0.8sec.

•  AnimprovedlearningalgorithmusingBDDlearnedthesamerulesin0.18sec.

Ø  Inoue,K.,RibeiroT.,Sakama,C.:“LearningfromInterpreta8onTransi8on”,MachineLearning,94(1):51-79,2014.

Ø  Ribeiro,T.,Inoue,K.,Sakama,C.:“ABDD-BasedAlgorithmforLearningfromInterpreta8onTransi8on”,Post-Proc.ILP2013,LNAI,Vol.8812,pp.47-63,2014.

“CanMachinesLearnLogics?”

InputAgentA(human/computer)Output

S:formulasLogicalSystemLT(⊆Th(S))

MachineM(S,T)LearningSystem CK

•  Giveninput(S,T),amachineMproducesanaxioma8csystemK.•  LFITcanlearnmeta-levelone-stepdeduc8onrules,e.g.,MP.Thescenario

canbeappliedtolearningabduc8onandothernon-standardlogics.Ø  Sakama,C.,Inoue,K.:“CanMachinesLearnLogics?”,AGI2015,LNAI,Vol.9205,pp.

341-351,2015.

1 2 6

3

4

pickup mov

e(4)

3

4

5 5a

5b 0.7

0.9

0.1

RevisingPlansinAdap8veSystems

EnvironmentSystem

Domainmodel

action

reaction

model revision

Behaviouralmodellearning/revisionthroughprobabilis@crulelearning

Ø  Sykes,D.,Corapi,D.,Magee,J.,Kramer,J.,Russo,A.,Inoue,K.:“Learningrevisedmodelsforplanninginadap8vesystems”,ICSE2013:63-71.

Ø  Marpnez,D.,Ribeiro,T.,Inoue,K.,Alenyà,G.,Torras,C.:“LearningProbabilis8cAc8onModelsfromInterpreta8onTransi8ons”,ICLP2015.

DNAsynthesis

p53

cyclinH

CAK

p21/WAF1

cdk2

cyclinE/cdk2

Rb

cdk7

DNAdamage

cyclinE

(1)(2)

(3)

(4)

(5)

(6)

promoted

promoted

suppressed

suppressed

(7)

suppressed

*

Cellcyclewithcyclin-dependentkinases(Schneideretal.,2002)

PathwayComple8onbyMeta-LevelAbduc8on

Ø  Inoue,K.,Doncescu,A.,Nabeshima,H.:“Comple8ngcausalnetworksbymeta-levelabduc8on,MachineLearning,91(2):239-277,2013.

Comple8ngSBGNNetworkswithGeneExpressionData(collabora8onwithLRI/Paris-Sud&INRA-CNRS)

Biologicalphenomena Formalsystem

Geneexpressiondata SBGN-AFnetwork

(Discre8zed)observa8ons

Discre8za8on

backgroundtheory

Causalnetworktransla8on

HypothesisNewinterac8onsorpredicates

Meta-LevelAbduc8onbySOLAR

Biologicalanalysisusingdatabaseor

literature

Inferenceengine

Ø  Yamamoto,Y.,Rougny,A.,Nabeshima,H.,Inoue,K.,Moriya,H.,Froidevaux,C.,Iwanuma,K.:“Comple8ngSBGN-AFNetworksbyLogic-BasedHypothesisFinding”,FMMB2014,LNBI,Vol.8738,pp.165-179,2014.

•  SBGN-AF:SystemsBiologyGraphicalNota8onAc8vityFlow

• Applica8ons:FSHR-inducedsignalingnetworks

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰Ac@on

Ac@onMachineLearning

DecisionMaking

DynamicTransi8on

TheDecisionMakingPartchoosesthebestnextac8onbysolvingconstraintsa8sfac8onproblems(CSP)andconstraintop8miza8onproblems(COP).

DynamicConstraintOp8miza8onProblems

•  Mostreallifeproblemsaredynamic,e.g.,transporta8on,teamforma8on,scheduling.

•  Thosemodelscanberepresentedas(dynamic)(hard&sou)constraintnetworks.

•  Goal:Minimizepenaltyandmaximizereward.•  Requiresfastcomputa8onofnewsolu8ons,yet

somequalityguaranteesshouldbeprovided.

DynamicCOP:Applica8onsandApproaches

•  TeamformaKon:Makingrobustteamsofagents.•  Nursererostering:Whenanurseisabsent,buildanewschedulewith

minimalandfairchanges.•  Timetabling:Reconstruct8metablesaccordingtositua8onchanges.

Ø  Okimoto,T.,Schwind,N.,Clement,M.,Ribeiro,T.,Inoue,K.,Marquis,P.:“HowtoFormaTask-OrientedRobustTeam”,AAMAS2015:395-403.

Ø  Clement,M.,Okimoto,T.,Schwind,N.,Inoue,K.:“FindingResilientSolu8onsforDynamicMul8-Objec8veConstraintOp8miza8onProblems”,ICAART2015:509-516.

Twodifferentapproaches:•  ProacKve:weprepareandtakeac8onsbefore

changeshappen.•  ReacKve:wedonotknowwhatwillhappen

andcanonlyreacttothechanges.

TaskOrientedRobustTeamForma8on•  Goal:Makeateamtoachieveagivensetoftasks

suchthatitcanbeeffec8veevenifsomeagentsbreakdown.

•  Ateamisk-robustifitcans8llperformalltasksauerlosinganykagents.

•  Robustnessisaproac8vesolu8ontoprepareforunpredictablechanges(thelossofsomeagents).

•  TORTFdoesnotprovemorecomputa8onallydemandingthanthetask-efficientteamforma8onproblem,i.e.,robustnessisforfree.

Ø  Okimoto,T.,Schwind,N.,Clement,M.,Ribeiro,T.,Inoue,K.,Marquis,P.:“HowtoFormaTask-OrientedRobustTeam”,AAMAS2015:395-403.

CyberSecurityTrade-OffProblem•  “Youcan'thave100percentsecurityandthenalsohave100percentprivacy

andzeroinconvenience.”(BarackObama,2013)•  Intercep8onandcommunica8onsdatareten8onmeasures,evenifthe

purposeissocialsecurity,areunderthedifficulttrade-offbetweensecurity,privacyandcost.”

•  Howtosolvethistrade-offandbuildthesocietalconsensus?•  WeusedMulK-ObjecKveConstraintOpKmizaKon(MO-COP)techniques.

PRIVACY

SECURITY COST

Ø  Okimoto,T.,Ikegai,N.,Ribeiro,T.,Inoue,K.,Okada,H.,Maruyama,H.:CyberSecurityProblemBasedonMul8-Objec8veDistributedConstraintOp8miza8onTechnique,WSR2013.

SelectedSolu8onsforMul8-Objec8veProblems•  Mo8va8on:ProductConfigura8onExample

→Many(some8mes,thousandsof)choicesaregiventotheuser.→Eachalterna8veinvolvesmanycriteria(e.g.,foracar,price,life8me,safety,brandreputa8on).→Impossibleforuserstochoosetheirpreferredproduct!

•  Characteriza8onofrepresentaKvesoluKons(Figureb)/Efficient

approxima8onprocedures(Figurec)•  Interes8ngbenefitsforanitera8veuse,forlarge-scaleproblems.Ø  Schwind,N.,Okimoto,T.,Konieczny,S.,M.,Inoue,K.:toappear,2016.

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰Ac@on

Ac@onMachineLearning

DecisionMaking

DynamicTransi8on

TheModelCheckingPartverifiesthattheagentworkscorrectlybypredic8ngfutureresultsindynamicenvironments.

ModelChecking

ResilienceofConstraint-BasedDynamicSystems

Ø  Schwind,N.,Okimoto,T.,Inoue,K.,Chan,H.,Ribeiro,T.,Minami,K.,Maruyama,H.:“SystemsResilience:AChallengeProblemforDynamicConstraint-BasedAgentSystems”,AAMAS2013,pp.785-788.

Ø  ReceivedThe3rdPrizeofBestChallengesandVisionsPapers.

•  Formaliza8onofgenericsystemsthroughCOP  variables:componentsofthesystem  constraints:interac8onsbetweenthesecomponents  opKmizaKonfuncKon:evaluatesthequalityofaconfigura8onofthesystem

•  Resistance+Recoverability+Func8onality=Resilience

BeliefPropaga8oninMul8-AgentSystems

•  Wewanttopredictthepropaga8onoffallaciousbeliefsinsocialnetworks(brandcrisismanagement,e.g.,Domino’sPizzacrisisinApril2009)

•  Wewanttotrackthetruthwhenseveralagentsdescribethesamesitua8onbuthaveconflic8ngbeliefs

Ø  Schwind,N.,Inoue,K.,Bourgne,G.,Konieczny,S.,Marquis,P.:“BeliefRevisionGames”,AAAI-15:1590-1596,2015.

•  Formaliza8onoftheframework,named"BeliefRevisionGames"•  Setofappealingproper8es:

–  3Preserva8onproper8es:Convergence,Responsiveness,Monotonicity

–  6agents’revisionpolicies(rangingfromcreduloustoskep8calones)

•  Inves8ga8onoftheextenttowhichtheseproper8esaresa8sfiedbytherevisionpolicies

•  Robustframework,consistentwithnaturalexpecta8ons:→credulousagentsaremoreresponsivethanskep8calones.→withskep8calagents,convergenceisguaranteed.

BeliefPropaga8oninMul8-AgentSystems

ResilienceProper8esinOscillatoryBiologicalSystems

Comet, etal.,“Simplified models for the mammalian circadian clock“,Evry Spring school on Modelling complex biological systems in the context of genomics, pp.85-106, 2012.

•  UseparametricKmePetrinetsinordertoanalyzepreciselythedynamicbehaviorofbiologicaloscillatorysystems,e.g.,themammaliancircadianclock.

•  Analyzeresilienceproper8esthatendorsemajorchangesintheirenvironment,e.g.jet-lags,day-nightalterna8ngwork-8me.

Ø  Andreychenko,A.,Magnin,M.,Inoue,K.:“ModelingofResilienceProper8esinOscillatoryBiologicalSystemsUsingParametricTimePetriNets”,CMSB2015:239-250.

Agent

DynamicEnvironments

History/Experience

𝑶

Goal𝑶

BackgroundKnowledge𝑩

Ac@on

Model𝑯

Time-SeriesData 𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰Ac@on

Ac@onMachineLearning

DecisionMaking

DynamicTransi8onModelChecking

Program/ContentAudiences

Viewer’sJourneyLog

𝑶

TargetGRP 𝑶

BackgroundKnowledgeDatabase 𝑩

ADDistribu@on

ADPlanningtoeachIP𝑯

Ra@ngDataExogenousData

𝑰

InternalState

𝑩∧𝑰∧𝑯⊨𝑶

?

OtherAgents𝑰AudienceReac@ons

ContentsProgramming

MachineLearning

DecisionMaking

DynamicTransi8onModelChecking

Adapta8ontoIPBroadcas8ngSystems

InferenceandLearninginDynamicEnvironments

AItechniquesthatcaniden8fythemodelofasystemindynamicenvironmentsandcanachievetasksevenwhen

unknownsitua8onsareencountered.

Modeling(MachineLearning)

DecisionMaking(ConstraintOp(miza(on)

Predic@on/Verifica@on(ModelChecking)

ResilientSystemsRobustness/Sustainability

DynamicSchedulingCyberSecurity

OpinionConstruc8onMul8-AgentLearning