0+`[ba8k-!a3 dc doe:p-01 -...

31

Upload: others

Post on 28-Feb-2021

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������� ��� � � ��� �� � � ���� ����� �

��� ���! "$#%'&)(�*� �,+.-0/ / �213 4�� �657�289+.:<;=��; &�%?>0@�&�%BA7@�&�%DC7@�&�%FE4@�&�%?>0>G(

HJI #�8K��;L#�-0M< N�1�:POQ;H R 89�! 4:P�S1TO ���;K+6�S17OHJU N�MVOL:PMW�YX0�S8 1��SO�Z�-08\[];H_^ �0+`[ba�8K-!a3�dc��dOe:P-01H_f :W � 4�S1 MW�YX0�S8 8Q�Sa�8Q��;L�S17Og�dOL:P-01h;H_i 5��d/ a�MP�0jlkm�0+.� � ��+.-!c01�:POL:V-01HJn 4op�d13+.�, Oe-0a�:W+S;

qsr tvuxwxy{z}|�u6~�tv�v�}ux~e���?|=y�ux�?y��}�W�?�Y�)�P�x�W�����������P�������P�?�P�\�}�)��y�w �Wuxtvt¡���)wD¢`|�£B¤¦¥e�vtvt���§B¨P¨Pq

Page 2: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ����������� ����� �� � ������� �

��������� �"!$# %�&(' )*���,+-/. !$&(#0��� ��12�43056%7�4�(8 3��9' ! : ;=<�<"> ��!?5@���A�-/. &(' BC!$#D�FE��(!$&(#0����� : >?<GIH- �J�K�(�(!?5@3L�4����� MC!$# �(!N&(#O�K� : >?<QP�RS- T 5@!$�(! #0!?5@��8K�(�43��4�K� 3��9' ! : ; > ��!?5@�K���- >?<�< �4�UEI!$#O!N��5@! ��3�!$M�� �"�V!?����WX3 �Y!$!N' Z4� [*! !$�(��&78�%

\ ' &�5]% M�)*#])QZ9Z4!$Z�5@��' M(&(30)Q3L�4���

^ #O��MC!N#O3��9!?�_�FE�)Q#O3L�a`C5@� )*Z��(!$&(#b)QZc�(!$30� d e . . Wf�hg]+-_i )Q��j �(!$&7#O���lkmZ4�n[Q! 3�%(#0!?��%(��Z � �Y12�4305]%(�4�(8 &(�(�930�-_i )Q��j 1D!$�48K%�3�!?� �4��3�!N#]5@�K�(�(!?5@3L�4����� )Q' ���(8 &7�(�430�-/o �48�%(Z4j MA)Q#])*Z4Z4!$Z pq�U� �Y3L#O�4B7&(3�!?� M(#0��5@!?�0�-/r ' M7%�)���� �s��� 3�&7�(�4�(8 1D!N�48�%�36� )Q&(3��K' )Q3�� 5N)QZ4Z4j

t4u vxwfyfzX{}|~w���vx�x�}wf�L����|Yz�wf��z��}� �����*�4�f� �����h� ���4�������4���4�b�}�*��z�y¡  wfvxv¢���*y¤£¥|�¦¨§l©L�xvxv��«ª¨¬4¬4t

Page 3: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

����� ��� � ������ ���� � ��������� � ����� ������

��� �"!"#%$'&)('*"&,+-*/. 01&32 45�6(7&,8-�69�:;01&<(>=@?>45$74 8-?A?B4C9�:D.FE�9�:,#"4C0G 4-HI+"HJ?K9ML (745�6(78-?A&,�"!"#"$ON

� P #%$7!"#"$A&<( 0�&<(B=�?B45$Q4 8-?A?>4C9R:E�9�:,#"4C0� P #%$7!"#"$A&<( 9 E-4M=@$78-? 8�S�E�9R:,#%4C(��TU8�(B(7&,V":,W �%8-&<(7W 0"9R$>9��XY8-?>2 8�S�$>9�?B+-4Z$AS[#"�\=@$7&,8-� &<(]#%�6^_�"8`LA���a #"2 9R� ?B4M9-0"9RV"&3:,&,$ W 8�S�?>4C(7#":,$ &)(]#"�%&,2 !b8-?>$>9R�c$

dfe 9R2 !":,4M(hg� ij!b454C=k* !%*"8-�"452 4 ?B4C=�8-+-�"&,$Q&,8-� lnm 9R&,Vb45:Do��� 2 9R+p4 =@:<9p(B(7&Dqb=59�$7&,8-� lsr 9R�69-014pt1u]9R:,#�vQ91t6w 8`LA:,45Wxo��X�&,�69��6=@&<9R:;!"?>4C01&<=�$7&,8-�

y,z {}|�~������s|���{}�}��|��Q��������|����I���<�������,���<���������I�,�I�����, �¡,¢K£�¤��¥�¦~¨§<|�{}{©¡�¤�~«ªO��¬®­/¯Q�}{}{�¡±°®²,²,y

Page 4: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��� ��� � ��� ����� ��� � ��� ��� ������ � !#"��

Sharp Left

SharpRight

4 Hidden Units

30 Output Units

30x32 Sensor Input Retina

Straight Ahead

$%$ &('*)*+-,/.0'�12&(3(4/'*165879.:+2'*;9+=</7>79?A@CB%D*E>FHGJILK�I=B%M=GNFOG%P9Q%RTS/UC3V+W)YX>'*&(&ZQNUC)\[].2^`_ba63(&(&8Qdc`e%e%$

Page 5: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��������������

w1

w2

wn

w0

x1

x2

xn

x0=1

.

.

Σ wi xi

n

i=0 1 if > 0

-1 otherwise{o =Σ wi xi

n

i=0

�����������������������! "##$##%

& ')(+*-,�. * �/�0� . 1�1�1�. * ����� 2 34 & 57698;:�<>=?'A@B:7C

D 57E :�6�'FE :G@H=I:KJMLFLON�@B: @�'FE P;LQ:�< R7:�ST6�57< U;576>VW69'F57U�X

�Y�[Z�0�+ "##$##%

& ')( Z* 1 Z� 2 34 & 576�8\:�<]=-'^@�:7C

_F` acbedefMgihjbTklacmcnibek9oqprhBflbesrfutip^prv�wyxFze{^|~}������uxF�u}�|�}F�r�F���i�ym�f�d��^beaca����yd���hl�����9mcacaq�����F�F_

Page 6: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ����������� �� ���������� ��� � � ����������������

x1

x2+

+

--

+

-

x1

x2

(a) (b)

-

+ -

+

� �! #"$�&%��!'�()% %�*,+ � -.%/�102-43�02-4'657(�89*,'6%:<; =6> (@?A�!8CB = ()% "$�D 4"$�E%/�!'�(FHGJI�KML�ION$PRQ S T U G2IVKWL�IONXPXY

Z[-#( %�*,+ � 0J-4'.57(�8C*,'.%\'4*](^")�! 4")�&%��!'�( >`_ 3C�: �,abB#adc4'#*,(@3C8C'4� > "$3Ce %/�D > " >`_ 3C�:\fg= �!"$�h0J*,")�,c4?A�,ij3C3�? > '�(@'4�!(k?A*,"Mlm%@*n0�( = �&%��,aoada

pCq rtsvuvwbxzy{s1|}rt~t�zsv|X���dy/w}sv�dwj�z�o�d���n�C�v�o��������j�C�j�����C�d�C���z�n~�w�u��osvrtr����nu �Wy}¡£¢¥¤X~trtr��§¦£qCqCp

Page 7: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� �������������� ������� ��� �� �������

����� ����� ���!�"$#&%'#

�(�*) +-,�.0/ 1�243*�5 "$#6%7#98:;.<) =�,?>3�2A@CB(D'E %7F9# D G�EIHKJ #:;1 @CB;L #6%NMO# L$D %'P9Q P J$DRL$JSD:T+ @CB BRU EIHKH MOP9Q BVD'E Q D , #9WXF$WCYZWC[ 2 M EIHKH #]\ ^K_a`Ibac$decgf b'`9h4_

iKj kmlonoprqtsulwvxkmymztlov|{~}�sVpxlo��pX�t}C}������K�o�C���������X�K�X�e���K���K���t��y�p�n��Clokmk��e��n��asx���¡ |ymkmk~�4¢�£K£K¤

Page 8: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� �������������� ������� ��� �� �������

����� � �"!$#&% ')( *+'),),�-.!/��#&%0�"1&%

24365 (7�8��')� '9� 1 : ��(;� '=<>,)'9� %?���",)@ <A%B�C���8��D ,)%

2 ���C: E <7FHG -.')%B��(7,)@ <AI ��,),

JLK MONQPQRTSLUVNXWYMOZO[LNQW]\_^`UARYNQa`RcbL^d^`e�f�g)hQidjlknmpo�mcg)qckrjsk)t`u)v8wLx�ZyRzP|{dNQMOM}urx�P�~�UY�����]ZOMOM_u6K��)�)�

Page 9: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��������� � � �������� �

��� ���������! #"%$&���('�)*�+�� #,-���.� /,10 2�34�.� 57698;:=<�> ?�8�6A@A'CBEDC�.�F�G H IKJML I NPO N�L Q.Q.QRL IKS*O�S

TM�.".U7 E34�V$&�%� IKW UX Y"#DZ$&" 0 ,4�C,40 ,4[.� "#DC� F\]�Z$&�F�^� �.�%�F�_�` a�bI c(d ef ghFi*jlkm hon G h=prq

s D����F� t ,- /�." ��uv"#�w$&,4��,4�Cx �*y�$&0 2C34�V

z4{ |~}X�X������}*��|~�~��}X�������/��}X�������-�������4�X�-���9�=����4�������4 �¡4¢w£�¤��¥�¦�¨§-}X|~|©¡�¤��«ª¬��­¯®±°��~|~|�¡³²¯´4´4µ

Page 10: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��������� � � �������� �

-1

0

1

2

-2-1

01

23

0

5

10

15

20

25

w0 w1

E[w

]

� ������������ ! " #�$%'&)( *+,.- "

- %'/�0 - "- % 120432323 - "

- %'56879: ���;���<���>= �@?>A���B

C $% D E'F ! " #�$%'&�GH��GJIC %'KLD E'F - "

- %MK

N�O PRQTSTUWVYXZQ\[]PR^R_YQT[a`cbdXeU]QTfdUHgYbJbdhji�k�lTmJnporqtsqHk�uHovnwo�xdy�z�{Y|�^}U~S��JQTPRP�yv|�S���X]�����a^RPRPcy��������

Page 11: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��������� � � �������� �

�������� �

������

���� ��!" �$# %��'&)(

��� � �

���� � !" �$# %��'& (

��� � � � !" �*# %��'& �

�+��� !," �*# %-�.&� � � !" �*# %��'& �

�+��� !," �*# /� 0 /1 �2&������ � � � � !" �*# %��'& ! # 1 �43 �5&

687 9;:=<=>@?BAC:5DE9;F;GB:=DIHKJ4AL>E:=M4>ONBJPJ4QSR�TVU=WPXZY\['][OTV^OY_X`YVa4bVcedBf�Fg>h<jiP:=9;9 b_f�<lk.AEmonqpIF;9;9KbsrotVtVu

Page 12: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��������� � � �������� �

� ����������� �"!$# �&%(')���*�,+-/.103254 254�6 7(8)039 :<;=7�>@?BA&CDFE3GIH JKLENMPO�M�ORQ S$TUE�V WYXZS M\[ E W�E�M�K ]_^ J`H�Sa^b]�KcVdfe8�?g-ihkjml HnSoKLS e8 M\[ JpH�S q�ScGrJ5]�K ]B^ M�O1W"stJuq�EvXwsxSI[ jENO<y - Mz[ J`HnS J5ENK{QUSrJm]�s�J|W<s�Jmq�EvXwsxS~} A M\[ J`H�SX�S�ENK�O�M�ORQ KLE@J5S ��S~}�QU} j }p�v�b��}

�������Z�_���N�Z���o� �r�@�$� �����B� �i�v� � �B� �N�Z�*�g�N���n�v� ���N�Z�����  ���B�Z�¡�B��� �B�o�L� �Z�����B�Z�@� �I�@���n�Z�_�Z�@� ����� �o�f¢�£ �¤ ���x�Z�B���N�¥�¦�o� �§�@�k� ¨ �F�Y�B� �o�o�L��©¤ ª �@� �r�@�$� d(e8«?k-ih �Z� -5.�03254*254�6 7(8<039 :<;`7�> ¢�£ �¬ �­��®������B��� �Z�"�i�L�����b� e8 �_� �_��� �����Z� ������b�v� ®����B� �B��� �@���_®���� ¯

¬ ª �@�F�r�@�$� �¥�Z���r�N� �����¥� °±�o�Z²@��� �F�`¢�£ �¨ �F�<³ ¨ �F�3´ A*+-«µ ¯ C58 �

¤ ª �@� �r�@�$� �Z�¥���r�N� ���x�Z�F°±�o�Z²@��� �F�p¢x£ ����)³ �F�¶´ ¨ ���

·Z¸ ¹»º½¼½¾À¿\ÁºIÃĹ»Å»Æ\º½Ã_ÇÉȦÁi¾Äº½Ê¦¾ÌË\È�ȦÍ~Î�ÏZнÑ�ÒÔÓPÕcÖÕÌÏZ×ÌÓ�ÒØÓZÙ¦ÚZÛgÜ\Ý�Å޾߼áà�º½¹»¹âÚ�Ý�¼�ã�ÁÄäæåtç_Å»¹»¹ÉÚ5èæéZéZê

Page 13: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

����� � ����

�� ����� ��������� ��������������� ���� �� ���!�"���"�#��� �%$ ��� &��!� ��� �'$ �)(

*,+ ���"�-������� �/.0�"1 �� ��'& �"��� -�����%�"�� �2 &3� �4�"����5� ��

* 6 �07 ����� �#�� �2 &31 �� � 8 ��%�"��������� ���"�9� :

;<�����%�"� ������� �9���"�-������� �=�> �� �!&3�'&?������$@��� �#� $@�%&���� �A�

* B �!�"���"�A�9� �%$ �9� �C���AD��E�=��� ��� F#2G������F��%&��H& IJ����F1 ������1 ��1 &=KL�!���=�%$ � �������

* B ��D�� � &��07 ����� �#�� �2 &31 �� � 8 ��%�"��������� ���"��� :

*NM D��E� IJF�� � �����"�����-��� $��"�O� �����#���"���!&J�����H&��

*NM D��E� IJF�� � �����"�����-��� $��"�O� ����� &�� �!�"���"5� �� 5#2 P

Q�R SUTWVWXZY\[]TC^_SU`Ua\TW^9bdce[3X_TWfeXhg\cHceij�k�lWmHnporqtsuqhk�vhownxo�yez�{�|\}�`~X�V��HTWSUS�zw}�V���[_�����9`USUSdz��������

Page 14: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

��������� ����� ��������������� �������� � ! �� �"#�$���� % �&�'�(����

) *+-,�. / 021�3 4 57698:<;>=&?A@ ;-BDCE;�=(?�F@ G HI=(?J:<KLBD6M?J:NBPOQ;�8RTS9U GTV WIHI?J; ?JXI; Y957698:<;>=&? Z [A\^]'_` ab(S _` c _` d e Z [ \ ]'_` af�gL,9h�3&/ 3&gi+-*kjl/ 0�1�3 4 5m6T8:<;�=(?l@ ;-BDCE;�=&?>F@ G HI=(?J:<KLBD6M?J:NBPOQ;�8npo G95A;-6TCqX ?J576M:<=I:<=kY ;srt6MV WkK<; u :<= vR9S9U G9V WIHk?J; ?JXk; YT5m698t:<;�=&? Z [Aw-]'_` ab&S _` c _` d e Z [Awx]'_` a

[A\y]'_` a2z Rb {wD|s\�}�~ w d � w��P�

[ w ]'_` a2z Rb }�~ w d � w ���

�m�i���D��� ��� ���T�#�A����������� ��� �s������� ��C�6M= 6MWIWk5�G�r&:<V 6M?J;� �T���s� �A�D������������� �s������� ��6M5D�I:<?�5m6M5D:<K<� CEK�G�B ;>K<� :¢¡ eV 698t; BJV 6MK<K�;�=IGTHIY9X

£<¤ ¥§¦©¨©ª¬«®­¯¦s°±¥§²§³®¦©°�´¶µ·­ ª±¦©¸·ªº¹®µ»µ·¼¾½�¿<À©Á»ÂÄÃÆÅ�Ç�ź¿<ȺÃÉÂÊÃ<Ë·Ì<Í7ήÏ�²ÐªÑ¨ÓÒ»¦©¥§¥ÔÌÉÏ�¨ÖÕ�­±×ÙØÛÚ�²§¥§¥¶Ì�ÜÙÝ<Ý<¤

Page 15: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������������ �� � ���� ������� ��� ����� � ���"! # $%�&���

F1 F2

head hid who’d hood... ...

'(' )+*-,-.0/213*�45)+6+72*-498;:<1=.5*-><.@?2:A:<BDCFE(G-HAIKJMLON�L@E(P@JQIRJ(S<T(UWV2XF6Y.Z,\[A*-)+)]TQXF,_^`15acbed96+)+);Tgfch(h(i

Page 16: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

������� ���� ����

w1

w2

wn

w0

x1

x2

xn

x0 = 1

.

.

net = Σ wi xii=0

n1

1 + e-neto = σ(net) =

����������� ����� ����� � ! ��" #%$�&(')����! &*

*�+ ,.-0/1 ��')� 2�34!025�634�8759;:=<?> /A@: / B �����C�)� * D �����C�E�F � '6GH& "I�634��J � � 3KG "L�M�N&O� "L�P')�N&O� 3Q$�RM�P�;��! ��3AGH�M&S T &U� ���M� � ! ��" $�&��M�S V WYX[Z]\]X_^H`ba6c d5aPZ]egfHcAh?i !j#k���M�0� ! ��" $U&��M�Q� lm G 'onp2�34!02(G.�bG.�E�M! &

qMr sutwvwxzy|{}t�~�su�u�|tw~E����{�x�tw��x��|�����?�H�M�w�������������M�������M���M�K�|�H��x�v¡ �twsus¢���Hv¤£o{�¥§¦Y¨E�usus��ª©§rMrM«

Page 17: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������� � ����� ����� ����� � �� ��� ��� � ����

� ��"!$# % ��&!$# '( )*,+.-�/ 0 *21 34*6587% '( ) *

��&! # / 0 *91 34*65 7% '(:) * ( / 0 *21 34*65 ��&! # / 0 *21 34*65% ) * / 0 * 1 3 * 5<;=>?1 � 34*�&!$#A@CBD% 1 ) *E/ 0 *�1 34*65 � 34*�&FHG 0 *

�&FHG 0 *�"! #I:JLKNMPO QSRUT�M V� 34*�&FHG 0 * %� W / FHG 0 *.5�"FHG 0 * % 3 * / ' 1 3 * 5

�"F�G 0 *�"! # % � /�X! Y XZ * 5�&! # % Z #\[ *] TUV ����"!$# % 1 )*^+.-_/ 0 *21 34*65?34* / ' 1 34*`5 Z #\[ *

acb dfehghikjmlCe.nodfpfqmehnsrut\lvioehw\iyxmtzt\{�|�}c~h�z���A�6� �y}c�y�����c�\�c���m��p�i�g��zehdfd�����g���lo�����spfdfdu����acac�

Page 18: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ����������� ������������ � ��������������

���! #"$ &%(') +*-, %�'#'/.0,1 #2�34"657"$8 5$9 %('#'�:;%(�=<>8?9 �4@A9 B�,-:C5-DE �4"F #'G5H%("$ &5JIK,L<NM=O 8PRQ 8�:S,L%?T63 "$:;%( #�A #�!2 ,VUW%(9 X!'#,�M=O 8Y DZ���!XA@A"S"$3A, "$:;%( #�A )�A2 ,[UW%(9 XA'#, "$8 "$3!, �!,-"�.08�:;\

%(�=< T]8?9 XA@A"$, "$3A, �A,-"^.08�:;\ 8�@!"$XA@A"H5_ D Q 8�: ,L%�TC3 8�@!"$XA@A" @!�A #" `

a-b c d�bLe Y f d�b]gheji�b f d�b]gk D Q 8�: ,L%�TC3 3A l<A<>,-� @!�A #" m

a-n c d(n�e Y f d(n-g ob6pVqsrutwvCrHt)xzy n|{}bCa-b

~ DZE X�<A%("F, ,�%�T63 �A,-"^.08�:;\ .0,- )2�34" yS� {��

y � {�� c y � {��/� � y � {��.S3A,-:H,

� yS� {�� � ��au�u� � {��

��� �����������w�V�����������F���+�������+� ���&�+¡Z¢�£#¤�¥&¦¨§ª©|«�© £#¬ §­¦®§#¯+°#±;²�³��´�µ�·¶&�����¸°­³��º¹h��»½¼�¾F������°¿�½�#�#À

Page 19: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ����� ��� � � ���������������������

� � ����� �"!$#&% �'!)(+*,!$#-%/.10�!$�/!$#-%2�3�+! 465)798;:=<�> ?@!$�"A B&%0 !)*C%2.��

�ED@� (2�"F"G A�!$#H!$�I�JF"�"KL!)� %M. �J�ONH�"%2�I�J�OG � �3�+!)*,%2!)� A ���=PHBQ(�ER �"F"FTSU#V� � F".&*$�JFXW�#H. %/#H!)*C!Y(+(O�J�O�"F"G A�F". NQ�JFT!$�O�+.��Z �"#H�"Z [HZ\ ] # PH�I� *C%M�^*C! WQ.=_`%M!$# ?@. �Iab(c?@!$F"Fed�*$�J# �O[H# Z [HF"%2�"PfF"!%2�"Z !)(Ig

� h _i%2!$# �"#Q*,F"[Q� ! ?@!$�"A B&% j :Jj 5$4Q7�k-j lm n/o^prq d`s�g�t uwv q2xUo^pyq{z l m n/o^prq d`s | }1g

��~ �"#f�"Z �"K$!Y(�!$�O�O. �/.10 !$� 79<O�=��4H��4b� !��'�JZ PHF"!Y(\ R �"F3F��"% A !$#f!$���=F"�"K$! ?@!$F"FT%2. (�[fNQ(�!Y��[H!$#-%!,�'�JZ PHF"!)(��

�������J�3#H�"#HA *L�J# %O� aJ! %2BH.�[Q(+�=#Q�H(�.=_��3%2!$�I�J%2�". #V( �(�F".�? �

�E� (��"#fA #f!$%�?@. �Ia �1_`%2!L� %2�I�J�"#H�"#fA �^(/0 !$�OG _X� (�%

�"� �������������, ¡��¢�£��� M¤¦¥§���¡��¨§�ª©�¥^¥§«­¬=®"¯�°^±³²�´�µ�´ª®"¶ª²�±·²"¸§¹"ºI»�¼=¢½�¾�À¿^�����Á¹�¼=�ÃÂÄ�¡ÅÇÆwÈM¢����¦¹ÊÉÇ�"�"Ë

Page 20: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ���������� � �� � ���� � ������� � ���������������������������

Inputs Outputs

� "!�#%$'&( *),+.-0/1 3254'-�6

7 -.8.+. 9 +. 38:+. ;=<'<'<'<'<><'< ? ;=<'<'<'<'<><'<<�;@<'<'<'<><'< ? <�;@<'<'<'<><'<<><�;@<'<'<><'< ? <><�;@<'<'<><'<<><'<�;@<'<><'< ? <><'<�;@<'<><'<<><'<'<�;@<><'< ? <><'<'<�;@<><'<<><'<'<'<�;=<'< ? <><'<'<'<�;=<'<<><'<'<'<'<�;@< ? <><'<'<'<'<�;@<<><'<'<'<'<><�; ? <><'<'<'<'<><�;

A !B- 3C.2EDGF�& H5&@!�#%-.&=I0JKJ

L5M NPORQRSUTWVXOZY[NP\P]WORY_^a`bVcS[ORdbSfeW`g`bhji�k5lRmgnporqts�qfk5ufovnwo5xby5z|{W}�\~S�Q��gORNPN�yv}�Q���V[�����_\PNPNay���L5L5�

Page 21: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ���������� � �� � ���� � ������� � ���������������������������

� "!$#&%('*),+.-Inputs Outputs

/ !102)3 "!14 57684"4�!9 :80<;*!$) )=!$>")=!<?@!9 A#302#B6C'* �-D ">7E"# F 68474�!9 G E"#B>"E"#

H 02:IE"!1?J1K*K*K*KLK*K*K M NPO*Q NRK2S NRK*O M J1K*K*K*KLK*K*KK�J1K*K*KLK*K*K M NPK�J N8J*J NRO*O M K�J1K*K*KLK*K*KK*K�J1K*KLK*K*K M NPK�J NRQ�T NVUWT M K*K�J1K*KLK*K*KK*K*K�J1KLK*K*K M NPQ*Q NRQ�T NXTYJ M K*K*K�J1KLK*K*KK*K*K*K�J<K*K*K M NPK*Z NRKW[ NRKWU M K*K*K*K�J<K*K*KK*K*K*K*K�J1K*K M N\U*U NRQ*Q NRQ*Q M K*K*K*K*K�J1K*KK*K*K*K*KLK�J1K M NPO*K NRK�J NRQ*O M K*K*K*K*KLK�J1KK*K*K*K*KLK*K�J M NP]*K NRQ2S NRK�J M K*K*K*K*KLK*K�J

^X_ `baPcPdfehgiakjl`bmbnhaPjporqsg@dlaPtsdvuhq8qswyx�zI{P|8}�~������vzI�v~�}�~I�s�I�,�h��m�d�c��8aP`b`�����c\��gl�V���pmb`b`r���V^I^I�

Page 22: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������������

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 500 1000 1500 2000 2500

Sum of squared errors for each output unit

�� ��������������� ���������� �!�"����#!�%$��&�!')(�*�+�,&-/.103240%*�5%.6-7.�8!9�:<;�=��>�?�A@&�� � B96=��DCE��FHGJI��� � 9LKH��M

Page 23: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������������

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 500 1000 1500 2000 2500

Hidden unit encoding for input 01000000

�� ��������������� ���������� �!�"����#!�%$��&�!')(�*�+�,&-/.103240%*�5%.6-7.�8!9�:<;�=��>�?�A@&�� � B96=��DCE��FHGJI��� � 9LKH��M

Page 24: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������������

-5

-4

-3

-2

-1

0

1

2

3

4

0 500 1000 1500 2000 2500

Weights from inputs to one hidden unit

�� ��������������� ���������� �!�"����#!�%$��&�!')(�*�+�,&-/.103240%*�5%.6-7.�8!9�:<;�=��>�?�A@&�� � B96=��DCE��FHGJI��� � 9LKH���

Page 25: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� �������������� �� ��� � �� �����������������������

� "!�#%$'&)(�* #%&,+.-/&0(1*2*43 +53�6 & 7'31-0!8796 $'(:$'6 ;<6=?> &) A@B!DCB+E(<3F*2GF7'3�HB!8796 $'(:$'6 ;<6 IJIKI=ML #<# 6 3F6 &0(�*4;<6= N *531-O@P!�+Q*4$K- GF R!�#�$'&0(�* #%&S+A-/&)(�*=MT R!8$U( 6 ;<7'*5$UC<7'& (<&)*.+2VW$'*5@ #%$YXZ&0 .&0(1*2$U(<$'*.!87�V[&)$'G�@�*O+

\ !8*5;: A& 3D]^-/3F(�_�&0 .G�&0(P-/&=?` (<$'*4$K!87'$Ka0& V[&0$'GF@�*.+ (<&,!8 Wa0&) A3=MT @:&0 A&b]c3� .&�d<$'(:$'*5$K!87�(<&0*eV[3� Rf+W(<&,!8 4gh7U$'(<&,!8 =?` (B-/ .&,!�+5$'(<GF7'i (<3F(�gh7U$'(<&,!8 ]c;:(B-/*5$'3F(B+MC�3j+A+5$'H<7'& !�+*5 R!D$'(<$'(<G C< A3FG� A&S+A+5&,+

k'l monqpqrtsvuwnyxzmo{o|vnqx4}�~JuQrznq�Jr��v~K~J���D�'�q�K�����������'�������'�J�'�"�v�D{�r�p��Knqmom����Dp���uz ¢¡¤£4{omom��¦¥¢k'k'§

Page 26: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ����������� ��� � ����������� ������ ��� � � � �

��� �"!$#&%(' )+*,'.-0/21$�3'.465798;: #=<?> @A� �3!$#&%B' ) *,'C-0/D1$�3' -6%(' @A# <?#6EF<?#&4D#6'�/D#&G @>',#6/IHJ�"<LK HM1$/DN 4D1$',O3!P# N,1QG,GR#6' !Q%S>3#6<

7 @,*,/MT 1$O3N�/�<U#&VW*,1$<U# #YXEA�"',#6'/21Q%(![Z 1$' '�*,T @A#6<9�B)1$',E,*,/\4^]JNF1QG,G_#6' *,'F1$/U4

` �3'/21$'*,�"*.4�) *F'.-0/D1$�"'.4a5798;: #=<?> @A�3*,'.GR#&G -0�"'/D1$'�*,�3*.49) *,'.-Y/D1$�3' -=%(' @A#%(E,E,<U�SX1$T %(/D#&G Hb1P/DN %(<U@,1$/D<^%(<U1$!$> 4DT %(!$!c#6<U<?�"<edR@>',#6/IHJ�"<LK HM1$/DN �3'F# NF1QG,G_#6' !Q%S>3#6< f ` > @A#6'.K(� g&h3i3h_jk �3<?'F1QK #=/ %B!�l gmh3i3h�n

7po '> )+*,'.-0/D1P�3' -6%(' @A# %(E,EF<?�qX1$T %(/D#mG /2� %B<?@,1P/D<L%B<?>%3-6-0*,<^%3-0> @> % ',#6/rHJ�3<LK HM1$/DN /rHJ� N,1sG,G_#6' !Q%S>3#6<^4f ` > @A#6'.KB� gmh3i3i�ntl

u$u vxwzyz{}|�~�wY��vx�x��wz�2����~�{�wz��{����Q���S�B�$�z�Q�����������$�������$���$�^�� B�¡{¢y¤£Qwzvxv¥�� By§¦¨~�©«ª­¬2�xvxv��t®«u$u$¯

Page 27: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ����������� ��� � � � � �

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

0.01

0 5000 10000 15000 20000

Err

or

Number of weight updates

Error versus weight updates (example 1)

Training set errorValidation set error

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 1000 2000 3000 4000 5000 6000

Err

or

Number of weight updates

Error versus weight updates (example 2)

Training set errorValidation set error

����� ��������� �!�#"$��%�& ��"('*)+�,�$��-+�/. )0)+1�243�5�60798;:=<>:/3�?/8@7A8�B+C�DFE G4%H�I�KJ0�����LC@G4��MN�$OQPSR(%����*CT�QU�U�V

Page 28: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ��� ����� � ��� ���� � ����� � ��������� ��������

... ...

��� �"! #$!$%&! %('*)+! ,�-

.�/102.�34652728:9<;

=�> -�?:@BAC�D?�E�-�,�! ?�F AC'G�H#

I*JLK A*@B@M,�%NAC!O� �P�HAC%(E�?�E�' )��HA*Q -SRT#$�*UVACEVQ %(�H@MR*'GE�?�WB?�E�' X Y�RZ�[Y]\ J �^A*@ �H#

_`/2_ acbed 9f82g b ; a 4ch b ;Oikjlgm9 be0 9on2j:jlprqZs�teu:vxwzy|{^yos�}ow[v~w��l�����2�Z4�9 d��:beaca �[� d`� g<�����O4 aca � _������

Page 29: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������� �� ��� � �� � ��� ����������

... ...

������� ���� !� #"%$&� ')(

*,+.-/*,0132545687:9

; �=<> #?��A@ B �DC�"%$&�!�

E�F (�C8GD<>��C�?)()'�� C�H <>"%�A�

$&����(JILKMKONPNQN RSGD�TRSGUH 'VRW�=@X'YKMZ ��[MH KA�\<MG��A�TR]$&�^H �

_`+�0 acbed 7W65f b 9 a 1cg b 9^hjikfl7 be- 7]m5i8iknporq�set8uwvyx{z\x]q�|]v}u~v��k�����5�r1�7 d��8beaca �}� d`� f:�����^1 aca � _������

Page 30: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������������ ���� � ������� ���������� ������

�����! #"%$'&�� "( )+*,� -.��$'*,/1032546 7�89 :<; => ?@+ACB ?DEACFHG+IKJLG3I'M 7�N DO@<P Q#DR@ :TSVU W ?X(Y[Z 9 SZ\X

]V)^ #$'� _,� 03 #)3*,��0 2`"'_,a��b2 ,2c-.��"%"d ,2fe� #"'gh�i2�46 7j89 :�; => ?@+ACB ?DEACFHG3IkJLG3I%M

lmmmn 7�N DR@oP QDO@ : S U p ?Z A Xrq JLG+I%Msttu`v N DO@v!w Z @ P v Q DO@vxw Z @

yKzz{ S}|�~~~�]c$'� 0`_�*,��0`/h��) -.��$%*,/�032�4� �,��*����h$%� a�/h_,�h��� � )+�i�C_,*,�h$'0�$'_,� �h��0T-._,)^�

�r�'� �����������K�C������������� ���¡����¢��¤£��(��¥�¦§'¨�©(ª¬«®­°¯�­¤§'±¤«²ª³«'´�µ'¶^·�¸�¹�º�¼»(�����½µ²¸�r¾¿��ÀÂÁÄÃ������ µÅ�ÂÆ'Æ'Ç

Page 31: 0+`[ba8K-!a3 dc dOe:P-01 - WPIweb.cs.wpi.edu/~cs4341/a17/LectureNotes/Slides/Mitchells...0+`[ba8K-!a3 dc dOe:P-01 H_f:W 4 S1 MW YX0 S8 8Q Sa8Q ;L S17Og dOL:P-01h; H_i 5 d/ aMP 0jlkm

� ������������� ����� ������

x(t) x(t) c(t)

x(t) c(t)

y(t)

b

y(t + 1)

Feedforward network Recurrent network

Recurrent network unfolded in time

y(t + 1)

y(t + 1)

y(t – 1)

x(t – 1) c(t – 1)

x(t – 2) c(t – 2)

(a) (b)

(c)

����� �������! #"$�&%'��(�)#��%+*-,."/�'��0.�21#,3,.4�57698�:3;=<?>A@B>269C2<D;E<9F.G9HJI#K7(L�M�ON3�����PGDK7��QR"'SUTWV+(����-GX�UY9Y9Z