1. Reducing the Sampling Complexity of Topic Models(KDD2014 Best Paper)Aaron Q Li (CMU)Amr Ahmed (Google)Sujith Ravi (Google)Alexander J Smola (CMU & Google)

2. KDD[22] KDD09: Efficient methods for topic model inferenceon streaming document collectionsby Limin Yao, David Mimno, Andrew McCallum (UMass)KDD08: Fast collapsed gibbs sampling for latent dirichletallocationby Ian Porteous, David Newman, Alex Ihler, Arthur Asuncion,Padhraic Smyth, Max Welling (UC Irvine) 3. : Collapsed Gibbs Sampling Walkers Alias Method Metropolis-Hastings Sampling 4. () ()BayesLDA (Latent Dirichlet Allocation)LDAPoisson-Dirichlet Process (PDP)Hierarchical Dirichlet Process (HDP) Walkers Alias Method (Walker aka )Rejection Sampling MCMC (Markov Chain Monte Carlo)(Collapsed) Gibbs SamplingMetropolis-Hastings SamplingLDA,PDP,HDPLDA 5. sneak peek = 6. sneak peek = 1 2 3 4 5 6 7 82, 4, 7, 7, 2, 8, 2, 4, LDAGibbs Sampling(!)() 7. sneak peek = 1 2 3 4 5 6 7 82, 4, 7, 7, 2, 8, 2, 4, LDAGibbs Sampling(!)() () O(1) Walkers alias method 8. sneak peek = 1 2 3 4 5 6 7 82, 4, 7, 7, 2, 8, 2, 4, LDAGibbs Sampling(!)() () O(1) Walkers alias method LDAGibbs Sampling 9. sneak peek = 1 2 3 4 5 6 7 82, 4, 7, 7, 2, 8, 2, 4, LDAGibbs Sampling(!)() () O(1) Walkers alias method LDAGibbs Sampling GibbsMetropolis-Hasting(MH)MH()Alias methodO(1)MHAccept/Reject 10. sneak peek = 1 2 3 4 5 6 7 82, 4, 7, 7, 2, 8, 2, 4, LDAGibbs Sampling(!)() () O(1) Walkers alias method LDAGibbs Sampling GibbsMetropolis-Hasting(MH)MH()Alias methodO(1)MHAccept/Reject = +mixture () 11. 1. 2. 1Dirichlet3. 2Walkers alias method (1974)O(1)4. / & 5. Latent Dirichlet Allocation(LDA)6. MCMCGibbs Sampling vs Metropolis-Hastings Sampling(non MCMCRejection Sampling)7. +Alias method+MH step 12. 1. 2. 1Dirichlet3. 2Walkers alias method (1974)O(1)4. / & 5. Latent Dirichlet Allocation(LDA)6. MCMCGibbs Sampling vs Metropolis-Hastings Sampling(non MCMCRejection Sampling)7. +Alias method+MH step 13. 1DirichletBernoulli()1231 2 3=(1,2,3)i := iDiscrete()i 0, i i =1 Z Discrete() Z Discrete()Z 1,2,3 with P(Z=i) = iDiscrete()Z{1,2,3} 14. 1DirichletBernoulli()1231 2 3=(1,2,3)i := iDiscrete()i 0, i i =1 Z Discrete() Z Discrete()Z 1,2,3 with P(Z=i) = iDiscrete()Z{1,2,3}Beta[0,1]http://www.ntrand.com/images/functions/plot/plotBeta.jpg () 15. 1DirichletBernoulli()1231 2 3=(1,2,3)i := iDiscrete()i 0, i i =1 Z Discrete() Z Discrete()Z 1,2,3 with P(Z=i) = iDiscrete()Z{1,2,3}Beta[0,1]http://www.ntrand.com/images/functions/plot/plotBeta.jpg ()[0,1]NN1 Dirichlet 16. 1DirichletDirichletBernoulli()1231 2 3=(1,2,3)i := iDiscrete()i 0, i i =1 Z Discrete() Z Discrete()Z 1,2,3 with P(Z=i) = iDiscrete()Z{1,2,3} Dir() :=(1,2,3) Dir(){: i 0, i i =1}123111 =(1,2,3) http://suhasmathur.com/2014/01/dirichlet-and-friends/Dir()=(1,2,3)i 0, i i =1 17. 2Walkers alias method (1974) k=51 2 3 4 51,5,5,3,5,4,5,5,3,1, 18. 2Walkers alias method (1974) k=51 2 3 4 51,5,5,3,5,4,5,5,3,1,1 2 3 4 5a b c d0 1u [0,1]if u < areturn 1else if u < breturn 2else if u < creturn 3else if u < dreturn 4elsereturn 5u O(k) O(log k) 19. 2Walkers alias method (1974) k=51 2 3 4 51,5,5,3,5,4,5,5,3,1,1 2 3 4 5a b c d0 1u [0,1]if u < areturn 1else if u < breturn 2else if u < creturn 3else if u < dreturn 4elsereturn 5u O(k) O(log k)O(1) (Walkers alias method)GNU Rver 2.2.0 20. 2Walkers alias method (1974) k=51 2 3 4 51,5,5,3,5,4,5,5,3,1,1 2 3 4 5a b c d0 1u [0,1]if u < areturn 1else if u < breturn 2else if u < creturn 3else if u < dreturn 4elsereturn 5u O(k) O(log k)O(1) (Walkers alias method)GNU Rver 2.2.0O(1) rand()%605+1Table(Alias Table)1 2 3 4 50 1A B C D E A: u0 p(x)/q(x)c cqxpi) y q, u [0,1]ii) If u 1/c p(y)/q(y)x y (y)Else i) q(x)p(x)q(x) c) p cdfcheck! 53. MCMC(1) Gibbs Samplingpxxpixix-i P(xi |x-i) t iid!p(burn-in)Coordinate-Descenthttp://zoonek.free.fr/blosxom//R/2006-06-22_useR2006_rbiNormGiggs.pnghttp://mikelove.files.wordpress.com/2008/09/gibbs.png 54. MCMC(2) Metropolis-Hastings SamplingpxGibbs Sampling q(y|x)y/i) , [0,1]ii) If thenElse()()q(y|x)!! (q(y)Gauss)q(y|x)p(y)(Gibbs100%Metropolis-Hastings) 55. 1. 2. 1Dirichlet3. 2Walkers alias method (1974)O(1)4. / & 5. Latent Dirichlet Allocation(LDA)6. MCMC (Markov Chain Monte Carlo)Gibbs Sampling vs Metropolis-Hastings Sampling(non-MCMCRejection Sampling)7. +Alias method+MH step 56. 1. 2. 1Dirichlet3. 2Walkers alias method (1974)O(1)4. / & 5. Latent Dirichlet Allocation(LDA)6. MCMC (Markov Chain Monte Carlo)Gibbs Sampling vs Metropolis-Hastings Sampling(non-MCMCRejection Sampling)7. +Alias method+MH step 57. key idea: Metropolis-Hasting-Walker SamplingMetropolis-Hasting-Walker Sampling pWalkers Alias methodO(1) p()pps t a l e O(1)()pMetropolis-Hasting Samplingp(pp)Gibbs SamplingMHW Sampling() 58. 1/2LDACollapsed Gibbs Samplingdiwdik()OK dkwkO(k) ()() 59. 2/2LDACollapsed Gibbs Sampling[22] !Sparse Sparse Densewd() ()[22] 60. 1/4() () dwSparse Dense[22] (k)(k) 61. 2/4Sparse Dense (k)(k)Metropolis-Hasting Sampling 62. 3/4mixtureMetropolis-Hasting SamplingsparseexactdenseAlias methodO(1)()MHxi) y ii) [0,1] u u min(1,) y x ! 63. 4/4Collapsed Gibbs ppdense2Alias Table qMetropolis-Hastings-Walker = qpMH pq(n=2, burn-inn=1) Alias Table(Alias Table) 64. () 65. ()LDAPDPHDP PubMedSmallNYTimesPDPHDP??AliasLDA () 66. () = 1 2 3 4 5 6 7 82, 4, 7, 7, 2, 8, 2, 4, LDAGibbs Sampling(!)() () O(1) Walkers alias method GibbsMetropolis-Hasting(MH)MH()Alias methodMHAccept/Reject = +mixture () 67. RC.P. & G., Springer, 2012.http://www.amazon.co.jp/dp/4621065270 A Theoretical and Practical Implementation Tutorial on Topic Modeling and GibbsSampling, W. M. Darling, 2011.http://u.cs.biu.ac.il/~89-680/darling-lda.pdf H24, , 2013.http://www.ism.ac.jp/~daichi/lectures/ISM-2012-TopicModels-daichi.pdf Statistical Machine Learning, Topic Modeling, and Bayesian Nonparametrics (DEIM2012 Tutorial), , 2011.http://www.slideshare.net/issei_sato/deim2012-issei-sato! Integrating Out Multinomial Parameters in Latent Dirichlet Allocation and NaiveBayes for Collapsed Gibbs Sampling, B. Carpenter, 2010.http://lingpipe.files.wordpress.com/2010/07/lda3.pdf (?) , 2014.http://www.amazon.co.jp/dp/427421530X