LatentDirichletAlloca/on

Blei,NgandJordan(2002)

PresentedbyDeepakSanthanam

WhatisLatentDirichletAlloca/on?•  Genera/veModelforcollec/onsofdiscretedata

– Datageneratedbyparameterswhichcanbelearnedandusedtodoinference.

LDAisahierarchicalBayesianModel

LDAandDocumentModeling

ADocumentofacollec/onismodeledasafinitemixtureoverunderlyingtopics.

Topicsinturnaremodeledasaninfinitemixtureoveranunderlyingsetoftopicprobabili/es.

Topicprobabili/esareexplicitrepresenta/onsofadocument.

Findshortdescrip/onsofmemberswhilepreservingsta/s/calrela/ons.

Documentclassifica/oniseasierwithLDA

PreviousSchemesforDocumentModeling

!–idfschemewherecountsaretakenforeachwordanddocumentismodeled.

LatentSeman.cIndexingwhichusesSVDtocaptureP‐idf

featureswhichcapturemostofthevariance.

pLSI–Eachwordinadocumentisasamplefromamixturemodelandgeneratedfromasingletopic.

(Eachdocumentisrepresentedasamixingpropor/onsoftopicsandthereisnotprobabilis/cmodelforthesepropor/ons)

AnEarlyExample..

w

Z

N

€

θ€

α

€

β

AnEarlyExample..

w

Z

N

€

θ€

α

€

β

Words

AnEarlyExample..

w

Z

N

Topics

Words€

θ€

α

€

β

AnEarlyExample..

w

Z

N

Topics

Words€

θ€

α

€

β

AnEarlyExample..

w

Z

N

Topics

Words€

θ€

α

€

β

Document

MixtureofTopics

AnEarlyExample..

w

Z

N

Topics

Words€

θ€

α

€

β

Document

MixtureofTopics

ExchangeabilityandBagofWords

Assump/onthattheorderofwordsinthedocumentcanbeneglected

AfinitesetofRandomVariables{x1,..xN}isexchangeableifthejointdistribu/onisinvarianttoanypermuta/onoftheseRVs.i.e.ifisapermuta/onof1toN:

e.g:Anyweightedaverageofi.i.dsequencesofrandom variablesisexchangeable.

€

P(x1,....,xN ) = P(xσ (1),...,xσ (N ))€

σ

€

σ

DeFine\’sTheorem

CanRewritetheJointofaninfinitelyexchangeablesequenceofRVsbydrawingarandomparameterfromsomedistribu/onandtrea/ngtheRVsasi.i.dcondi/onedonthatrandomparameter.

Zn

N

€

θ

DeFine\’sTheorem

CanRewritetheJointofaninfinitelyexchangeablesequenceofRVsbydrawingarandomparameterfromsomedistribu/onandtrea/ngtheRVsasi.i.dcondi/onedonthatrandomparameter.

Zn

N

€

θRandomParameterofaMul/nomialovertopics

DeFine\’sTheorem

CanRewritetheJointofaninfinitelyexchangeablesequenceofRVsbydrawingarandomparameterfromsomedistribu/onandtrea/ngtheRVsasi.i.dcondi/onedonthatrandomparameter.

Zn

N

€

θRandomParameterofaMul/nomialovertopics

Topicsarenowi.i.dcondi/onedontheta.

LDAandExchangeability

•  Wordsaregeneratedbytopicswithafixedcondi/onaldistribu/on

•  Topicsareinfinitelyexchangeablewithinadocument.

•  ForadocumentW=(w1,w2,..wN)ofNwordsandacorpusofMdocumentsC={W1,W2,…WM}forktopicsdenotedbyz,

€

p(w,z) = p(θ) p(zn |θ)n=1

N

∏ p(wn | zn )

d(θ)∫

Whattypeofdistribu/oncanbeusedtomakeiteasyforinference?

TheDirichletDistribu/on

•  AK‐DimensionalDirichletRVcantakevaluesinthe(k‐1)simplexandhasthefollowingdensityonthatsimplex

Whereisak‐vectorwithcomponentsgreaterthan0.

Dirichletmakesiteasyforinferenceasithasfinitedimensionalsufficientsta/s/csandisaconjugatetotheMul/nomialdistribu/on.

€

θ

€

α

Genera/veProcessofLDA

•  Choose•  Choose•  Foreachwordwn:– chooseatopiczn~Mul6nomial()

– Chooseawordwnfromamul6nomialprobabilitycondi6onedonthetopiczn

– BetaisakxvMatrixand

€

N ~ Poisson(ξ)

€

θ ~ Dir(α)

€

θ

€

p(wn | zn,β)

€

βij = p(w j =1 | zi =1)

GraphicalModelofLDAThejointoverthetopicsandwordsisgivenby,

SampledonceeverywordSampledonceeverydocument

Sampledoncepercorpus

TheMarginalofaDocumentandTheProbabilityoftheCorpus.

•  Integra/ngoverthetopicmixturesandsummingoverthewordsgivestheMarginalofadocument.

•  ProductoftheMarginalsofalldocumentsgivestheprobabilityofthecorpus

CorpuslevelDocumentLevelWordLevel

GeometricRepresenta/on

InferenceProblem

WehavetofindthePosteriorofthelatentvariablesofadocument.

Intractablecauseweneedtomarginalizeoverhiddenvariables.

UseapproximateinferencelikeMCMCorvaria.onalmethods.

TightCouplingbetweentwoparameters

Varia/onalInference

‐Dropedgeswhichcausethecouplingingraphicalmodel.

‐Simplifiedgraphicalmodelwithfreevaria/onalparameters ‐Problema/ccouplingnotpresentinthesimplergraphical

model.

Varia/onalInference

‐Dropedgeswhichcausethecouplingingraphcialmodel.

‐Simplifiedgraphicalmodelwithfreevaria/onalparameters ‐Problema/ccouplingnotpresentinthesimplergraphical

model.

Problema/cedge

Varia/onalInference Resultsinthefollowingdistribu/on:

MinimizetheKullback‐Leiblerdivergence.

Equa/ngderiva/vesofKLtozero,wegettheupdateequa/ons,

Varia/onalInference Resultsinthefollowingdistribu/on:

MinimizetheKullback‐Leiblerdivergence.

Equa/ngderiva/vesofKLtozero,wegettheupdateequa/ons,

DirichletParameter Mul/nomialParameter

ParameterEs/ma/onUsingempiricalBayes

‐Findtheparameterswhichmaximizetheloglikelihoodofdata.

‐Intractableforsamereasons. ‐Varia/onalinferenceprovideda/ghtlowerbound.

Alterna/ngVaria/onalExpecta/onMaximiza/on:

‐E‐Step:foreachdocumentfindop/mizingvaluesof varia/onalparameters.

‐M‐Step:Maximizethelowerboundonthelikelihoodwith

respecttothemodelparameters.€

(γ,φ)

€

(α,β)

Smoothing

•  Likelihoodofpreviouslyunseendocumentsisalwayszero.

•  SmoothmatrixbyconsideringitselementsasRVswithaposteriorcondi/onedondata.

•  Dothewholeinferenceprocedureagainfornewmodeltogetnewupdateequa/ons.€

β

AnotherDirichletPrior

TreatElementsofBetaasRVsendowedwithaposterior

Smoothing

•  Likelihoodofpreviouslyunseendocumentsisalwayszero.

•  SmoothmatrixbyconsideringitselementsasRVswithaposteriorcondi/onedondata.

•  Dothewholeinferenceprocedureagainfornewmodeltogetnewupdateequa/ons.€

β

FinalHyperparameters

ExtendingLDA

•  Makeacon/nuousvariantusinggaussiansinsteadofmul/nomials.

•  Par/cularformofclusteringbyhavingamixtureofDirichletdistribu/onsinsteadofone.

•  WhatmustbedonetoextendLDAtoamoreusefulmodel?

•  CanweusethisLDAmodelinComputerVision?

Applica/oninComputerVisionOneofthemethodsextendedinDescribingvisualscenes(Sudderthetal2005)

Topicsinascene

LlamaSkyTreeGrass

LlamaLlamaSkyTreeGrass

Applica/oninComputerVisionOneofthemethodsextendedinDescribingvisualscenes(Sudderthetal2005)

Topicsinascene

LlamaSkyTreeGrass

Spa/alrela/onships?

LlamaLlamaSkyTreeGrass

Applica/oninComputerVisionOneofthemethodsextendedinDescribingvisualscenes(Sudderthetal2005)

Topicsinascene

LlamaSkyTreeGrass

MoreHierarchies..CoolerModels..

LlamaLlamaSkyTreeGrass

Evencoolermodels..!

Takehomemessage.

LDAillustrateshowProbabilis/cmodelscanbescaledup.

Withgoodinferencetechniques,wecansolvehardproblemsinmul/pledomainswhichhaveamul/plehierarchies.

Genera/vemodelsaremodularandextensibleeasily.

ThankYou!