brainlikebrainlike cognitive computation: cognitive...

BrainlikeBrainlike Cognitive Computation: Cognitive Computation: Past, Present, and FuturePast, Present, and Future

한국뇌공학회 창립 심포지움KAIST, 2012. 3. 16

장 병 탁Byoung-Tak Zhang

서울대학교 컴퓨터공학부 &서울대학교 컴퓨터공학부 &인지과학, 뇌과학 협동과정 겸임

http://bi.snu.ac.kr/

OutlineOutlineOutlineOutline

Wh I B i S i d E i i (뇌공학)? What Is Brain Science and Engineering (뇌공학)? From Brain to Engineering (and AI) From Engineering (and AI) to Brain

History of Brain-Style Computation Neural Networks Connectionist Cognitive Science Connectionist Cognitive Science Neural Information Processing Machine Learning Neuroinformatics Neuroinformatics Computational Neuroscience Computational Cognitive Neuroscience

Toward Brainlike Cognitive Computation Toward Brainlike Cognitive Computation Artificial Cognitive Systems Cognitive Robotics

C i i C i

(c) 2009-2010 SNU Biointelligence Laboratory, http://bi.snu.ac.kr/

2

Cognitive Computing

TuringTuring’’s Dream of Intelligent Machiness Dream of Intelligent Machinesgg gg

Alan Turing(1912-1954)

© 2006, SNU Biointelligence Lab, http://bi.snu.ac.kr/

3Computing Machinery and Intelligence (1950)

Von Neumann’s Von Neumann’s The Computer and the The Computer and the Brain Brain (1958)(1958)

4(c) 2000-2007 SNU CSE Biointelligence Lab, http://bi.snu.ac.kr/

John von Neumann (1903-1957)

The Brain vs ComputerThe Brain vs ComputerThe Brain vs. ComputerThe Brain vs. Computer

1. 10 billion neurons 2. 60 trillion synapses

1. Faster than neuron (10-9 sec)cf. neuron: 10-3 secy p

3. Distributed processing4. Nonlinear processing5 P ll l i

f3. Central processing4. Arithmetic operation (linearity)5 S ti l i5. Parallel processing 5. Sequential processing

From Biological Neuron to From Biological Neuron to o o og ca Neu o to o o og ca Neu o to Artificial NeuronArtificial Neuron

Dendrite Cell Body AxonDendrite Cell Body Axon

Neural Network (MLP)Neural Network (MLP)Error Backpropagation

Neural Network (MLP)Neural Network (MLP)E

Output Comparison

Information Propagation otwE 2)(1)(i

iiii wEwwww

,

p g

Input x1 Weights

outputsk

kkd otwE )(2

)(

Input x2 Outputx )(xfo

Input x3

Activation FunctionScaling FunctionInput Layer Hidden Layer Output Layer

Activation Function

© 2010, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 7

Past and PresentPast and Present

8

History of BrainHistory of Brain--Style Computation (1/2)Style Computation (1/2)History of BrainHistory of Brain Style Computation (1/2)Style Computation (1/2)

1943 McCulloch & Pitts Neuron 1948 Hixon Symposium on Cerebral Mechanisms in Behavior 1949 Hebbian Learning & Cell Assembly

1955 Session on Learning Machines (Selfridge’s Pandemonium) 1957 Perceptron (Rosenblatt) 1958 The Computer and the Brain (von Neumann) 1960 Adalines & Madalines (Widrow) 1970’s Kohonen, Amari, Grossberg 1979 First Issue of Cognitive Science Journal 1979 First Conference on Cognitive Science 1980 First Machine Learning Workshop (ML 1)

1982 Parallel Models of Associative Memory (Hinton & Anderson) 1982 Connectionist Models and Their Properties (Feldman & Ballard) 1982 Hopfield Networkp 1984 Kohonen Network (SOM) 1986 Machine Learning Journal (Langley)

(c) 2012 SNU Biointelligence Laboratory, http://bi.snu.ac.kr/

9

History of BrainHistory of Brain--Style Computation (2/2)Style Computation (2/2)History of BrainHistory of Brain Style Computation (2/2)Style Computation (2/2)

1986 Parallel Distributed Processing (PDP) Models (Rumelhart & McClleland) 1987 International Neural Network Society (INNS) 1988 Connectionist Models Summer School (Touretzky, Hinton, Sejnowsky) 1988 Bayesian Networks (Pearl) 1989 Neural Computation Journal (Sejnowsky) 1989 First Neural Information Processing Systems Conference (NIPS 1) 1990 Brain Style Computation: Learning and Generalization (Rumelhart)

1992 First Computational Neuroscience Conference (CNS) 1992 Support Vector Machines (Vapnik) 1993 First ICML Conference (ML 10, Int. Conf. on Machine Learning) 1995 Helmholtz Machine (Hinton, Dayan, Neal)

2002 EU FP6 Artificial Cognitive Systems 2006 Deep Belief Networks (DBN) 2006 First Computational Cognitive Neuroscience Conference (CCN)p g ( ) 2008 IBM Cognitive Computing Project (SyNAPSE) 2009 MIT Intelligence Initiative (I2)


10

From Neural Nets to Machine LearningFrom Neural Nets to Machine LearningFrom Neural Nets to Machine LearningFrom Neural Nets to Machine Learning

Symbolic Learning Probabilistic Learning Symbolic Learning Version Space Learning Case-Based Learning

Probabilistic Learning Bayesian Networks Helmholtz Machines Hidden Markov Models

Neural Learning Multilayer Perceptrons Self-Organizing Maps

Hidden Markov Models Markov Random Fields Latent Variable Models

H t kg g p Support Vector Machines Kernel Machines

Evolutionary Learning

Hypernetworks Deep Belief Networks

Other Methods Evolutionary Learning

Evolution Strategies Evolutionary Programming

Decision Trees Reinforcement Learning Boosting Algorithms

Genetic Algorithms Genetic Programming Molecular Programming

g g Mixture of Experts Independent Component Analysis

© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/11

g g

장병탁, 차세대 기계학습 기술, 정보과학회지, 25(3), 2007

FutureFuture

12

Towards HumanTowards Human--Level IntelligenceLevel IntelligenceggCreative Uncertain

Adaptive Inattentive

Sociable Emotional

Versatile Illogical

To achieve a true human-level intelligence, brain-like information

1 + 2 = 5 !100 < 10 ?

2012-12-06 13

To achieve a true human level intelligence, brain like information processing is required.

Brain as Widely Distributed, Parallel, Interactive, Brain as Widely Distributed, Parallel, Interactive, Overlapping, Dynamic Relational Memory NetworksOverlapping, Dynamic Relational Memory NetworksOverlapping, Dynamic Relational Memory NetworksOverlapping, Dynamic Relational Memory Networks

[Fuster, 2004]

15Hagmann et al., 2010

HypernetworksHypernetworks: Brain: Brain--Style Learning NetsStyle Learning NetsHypernetworksHypernetworks: Brain: Brain Style Learning NetsStyle Learning NetsAssembly Codes

Encoding

3. Module Discovery


16

Decoding

HypernetworkHypernetwork as a Probabilistic Model of as a Probabilistic Model of DistributedDistributed Parallel Associative MemoryParallel Associative Memory

x1 xxxXWSXH

I,...,, )( ),,(

as definedisrk hypernetwoThe

21[Zhang, DNA-2006]

[Zhang, IEEE CIM, 2008]

Distributed Distributed Parallel Associative MemoryParallel Associative Memory

x1x2

x3 x14

x15

K

iii

i

WWWW

SkXSSS

)()3()2(

tT i i),...,,(

|| , ,

[Zhang, IEEE CIM, 2008]

x4 x13 11rkhypernetwo theofenergy The

NnD 1)( }{ :setTraining

x

321

32132121

2121

)()(

,,

)()()()3(

,

)()()2()(

)];(exp[1 )|(

ondistributiy probabilit The

...61

21 );(

iiiiiiiiii

nn

iii

nnn

ii

nnn

WEWP

xxxwxxwWE

xx

x

x5 x12

321321321

212121

,,

)()()()3(

,

)()()2(

11

...61

21exp

)Z(1

)];(p[)Z(

)|(

iiiiiiiiii

K

iii

nnn

ii

nn xxxwxxwW

W

x6 x11

2121...21

)()()()(

2 ,...,,

)()()()(

1exp)Z(

isfunction partition thewhere

,...)(

1exp)Z(

1 k

kiiikiii

K mmmk

k iii

nnnk

xxxwW

xxxwkcW

© 2009, SNU Biointelligence Lab, http://bi.snu.ac.kr/

17

x7

x8 x9

x10

)( 21

21...212 ,...,,...

)(exp)Z(

m kkiiikiiik iii

xxxwkc

Wx

Reading Brain by (BrainReading Brain by (Brain Style) Machine LearningStyle) Machine Learning

[Nature Reviews Neuroscience, 2006]

Reading Brain by (BrainReading Brain by (Brain--Style) Machine LearningStyle) Machine Learning

Reading the Mind (Thought, Perception) of Humans by Machine Learning of Brain Activitiesg ( g , p ) y g

Application: Lie Detector Application: Lie Detector

© 2010, SNU Biointelligence Lab, http://bi.snu.ac.kr/18

Toward Toward BrainlikeBrainlike Cognitive ComputationCognitive ComputationToward Toward BrainlikeBrainlike Cognitive ComputationCognitive Computation

Artificial Cognitive gSystems

Cognitive Robotics

Cognitive Computing


Appendix:Appendix:ppppBrain Science and Engineering atBrain Science and Engineering atSNUSNUSNUSNU

20

SNU PSNU PSNU ProgramsSNU Programs

1996.3 - : Cognitive Science Program (Graduate) 2001.3 - : Brain Science Program (Graduate)g ( ) 2009.3 - : Brain, Mind, and Behavior Program

(Undergraduate)( g ) 2009.9 - : WCU Brain and Cognitive Sciences

Program (Graduate)Program (Graduate)

인문대(철학, 언어학), 사회대(심리학, 경제학), 사범대(교육학) 대(철학 어학) 사회대(심리학 경제학) 사범대( 육학)자연대(신경과학), 의대(신경생리학), 약대(신경약리학)공대(컴퓨터, 전기전자, 화학생물), 농생명대(생명과학),예술대(인지음악), 체육대(지각행동)


brainlikebrainlike cognitive computation: cognitive...

Documents