artificial neural networks (ann)
DESCRIPTION
CES 514 Lec 11 April 28,2010 Neural Network, case study of naïve Bayes and decision tree, text classification. Artificial Neural Networks (ANN). Output Y is 1 if at least two of the three inputs are equal to 1. Neural Network with one neuron. - PowerPoint PPT PresentationTRANSCRIPT
CES 514 Lec 11 April 28,2010
Neural Network, case study of naïve Bayes and decision tree, text
classification
Artificial Neural Networks (ANN)
X1 X2 X3 Y1 0 0 01 0 1 11 1 0 11 1 1 10 0 1 00 1 0 00 1 1 10 0 0 0
X1
X2
X3
Y
Black box
Output
Input
Output Y is 1 if at least two of the three inputs are equal to 1.
Neural Network with one neuron
X1 X2 X3 Y1 0 0 01 0 1 11 1 0 11 1 1 10 0 1 00 1 0 00 1 1 10 0 0 0
X1
X2
X3
Y
Black box
0.3
0.3
0.3 t=0.4
Outputnode
Inputnodes
otherwise0
trueis if1)( where
)04.03.03.03.0( 321
zzI
XXXIY
Rosenblatt 1958(perceptron) (also known as threshold logic unit)
Artificial Neural Networks (ANN)
Model is an assembly of inter-connected nodes and weighted links
Output node sums up each of its input value according to the weights of its links
Compare output node against some threshold t
X1
X2
X3
Y
Black box
w1
t
Outputnode
Inputnodes
w2
w3
)( tXwIYi
ii Perceptron Model
)( tXwsignYi
ii
or
Training a single neuron
Rosenblatt’s algorithm:
Linearly separable instances
Rosenblatt’s algorithm converges and finds a separating plane when the data set is linearly separable.
Simplest example of instance that is not linearly separable:
exclusive-OR (parity function)
Classifying parity with more neurons
A neural network with sufficient number of neurons can classify any data set correctly.
General Structure of ANN
Activationfunction
g(Si )Si Oi
I1
I2
I3
wi1
wi2
wi3
Oi
Neuron iInput Output
threshold, t
InputLayer
HiddenLayer
OutputLayer
x1 x2 x3 x4 x5
y
Training ANN means learning the weights of the neurons
Algorithm for learning ANN
Initialize the weights (w0, w1, …, wk)
Adjust the weights in such a way that the output of ANN is consistent with class labels of training examples– Objective function:
– Find the weights wi’s that minimize the above objective function e.g., backpropagation algorithm
details: Nillson’s ML (Chapter 4) PDF
2),( i
iii XwfYE
WEKA
WEKA implementation
WEKA has implementation of all the major data mining algorithms including:
• decision trees (CART, C4.5 etc.)• naïve Bayes algorithm and all variants• nearest neighbor classifier• linear classifier• Support Vector Machine • clustering algorithms• boosting algorithms etc.
Weka tutorials
http://sentimentmining.net/weka/
Contains videos showing how to use weka for various data mining applications.
A case study in classification
CES 514 course project from 2007 (Olson)
Consider a board game (e.g checkers, backgammon). Given a position, we want to determine how strong the position of one player (say black) is.
Can we train a classifier to learn this from training set?
As usual, problems are: • choice of attributes• creating labeled samples
Peg Solitaire – one player version of checkers
• To win, player should remove all except one peg.• A position from which a win can achieved is called a solvable position.
Square board and a solvable position
Winning move sequence: (3, 4, 5), (5, 13, 21), (25, 26, 27), (27, 28, 29), (21, 29, 37), (37, 45, 53), (83, 62, 61), (61, 53, 45)
How to choose attributes?
1. Number of pegs (pegs).2. Number of first moves for any peg on the
board (first_moves).3. Number of rows having 4 pegs separated by
single vacant positions (ideal_row).4. Number of columns having 4 pegs separated
by single vacant positions (ideal col).5. Number of the first two moves for any peg
on the board (first_two).6. Percentage of the total number of pegs in
quadrant one (quad_one).7. Percentage of the total number of pegs in
quadrant two (quad_two).
List of attributes
•Percentage of the total number of pegs in quadrant three (quad_three).
•Percentage of the total number of pegs in quadrant four (quad_four).
•Number of pegs isolated by one vacant position (island_one).
•Number of pegs isolated by two vacant positions (island_two).
•Number of rows having 3 pegs separated by single vacant positions (ideal_row_three).
•Number of columns having 3 pegs separated by single vacant positions (ideal_col_three).
Summary ofperformance
Text Classification
• Text classification has many applications– Spam email detection– Automated tagging of streams of news articles, e.g.,
Google News– Online advertising: what is this Web page about?
• Data Representation– “Bag of words” most commonly used: either counts or
binary– Can also use “phrases” (e.g., bigrams) for commonly
occurring combinations of words
• Classification Methods– Naïve Bayes widely used (e.g., for spam email)
• Fast and reasonably accurate– Support vector machines (SVMs)
• Typically the most accurate method in research studies• But more complex computationally
– Logistic Regression (regularized)• Not as widely used, but can be competitive with SVMs (e.g., Zhang and Oles, 2002)
Types of Labels/Categories/Classes
• Assigning labels to documents or web-pages– Labels are most often topics such as Yahoo-categories– "finance“,"sports,"news>world>asia>business"
• Labels may be genres– "editorials" "movie-reviews" "news”
• Labels may be opinion on a person/product– “like”, “hate”, “neutral”
• Labels may be domain-specific– "interesting-to-me" : "not-interesting-to-me”– “contains adult language” : “doesn’t”– language identification: English, French, Chinese, …
Ch. 13
Common Data Sets used for Evaluation
• Reuters– 10700 labeled documents – 10% documents with multiple class labels
• Yahoo! Science Hierarchy – 95 disjoint classes with 13,598 pages
• 20 Newsgroups data– 18800 labeled USENET postings– 20 leaf classes, 5 root level classes
• WebKB– 8300 documents in 7 categories such as “faculty”, “course”, “student”.
Practical Issues• Tokenization
– Convert document to word counts = “bag of words”– word token = “any nonempty sequence of characters”– for HTML (etc) need to remove formatting
• Canonical forms, Stopwords, Stemming – Remove capitalization – Stopwords:
• remove very frequent words (a, the, and…) – can use standard list
• Can also remove very rare words, e.g., words that only occur in k or fewer documents, e.g., k = 5
• Data representation– e.g., sparse 3 column for bag of words: <docid termid count>
– can use inverted indices, etc
challenges of text classification
M.L classification techniques used for structured data
Text: lots of features and lot of noise No fixed number of columns No categorical attribute values Data scarcity Larger number of class label Hierarchical relationships between
classes less systematic unlike structured data
Techniques Nearest Neighbor Classifier
•Lazy learner: remember all training instances•Decision on test document: distribution of labels
on the training documents most similar to it•Assigns large weights to rare terms
Feature selection•removes terms in the training documents which
are statistically uncorrelated with the class labels Bayesian classifier
•Fit a generative term distribution Pr(d|c) to each class c of documents .
•Testing: The distribution most likely to have generated a test document is used to label it.
Stochastic Language Models
Model probability of generating strings (each word in turn) in a language (commonly all strings over alphabet ∑). E.g., a unigram model
0.2 the
0.1 a
0.01 man
0.01 woman
0.03 said
0.02 likes
…
the man likes the woman
0.2 0.01 0.02 0.2 0.01
multiply
Model M
P(s | M) = 0.00000008
Sec.13.2.1
Stochastic Language Models
Model probability of generating any string
0.2 the
0.01 class
0.0001 sayst
0.0001 pleaseth
0.0001 yon
0.0005 maiden
0.01 woman
Model M1 Model M2
maidenclass pleaseth yonthe
0.00050.01 0.0001 0.00010.2
0.010.0001 0.02 0.10.2
p(s|M2) > p(s|M1)
0.2 the
0.0001 class
0.03 sayst
0.02 pleaseth
0.1 yon
0.01 maiden
0.0001 woman
Sec.13.2.1
Using Multinomial Naive Bayes Classifiers to Classify Text: Basic
method
Attributes are text positions, values are words.
too many possibilities Assume that classification is independent of the positions of the words
Use same parameters for each position Result is bag of words model (over tokens)
)|text""()|our""()(argmax
)|()(argmax
1j
j
jnjjCc
ijij
CcNB
cxPcxPcP
cxPcPc
Sec.13.2
Textj single document containing all docsj
for each word xk in Vocabulary nk number of occurrences of xk in Textj
Naive Bayes: Learning
From training corpus, extract vocabulary Calculate required P(cj) and P(xk | cj) terms
For each cj in C do docsj subset of documents for which the target class is cj
||)|(
Vocabularyn
ncxP k
jk
|documents # total|
||)( j
j
docscP
Sec.13.2
Naive Bayes: Classifying
positions all word positions in current document which contain tokens found in Vocabulary
Return cNB, where
positionsi
jijCc
NB cxPcPc )|()(argmaxj
Sec.13.2
Naive Bayes: Time Complexity
Training Time: O(|D|Lave + |C||V|)) where Lave is the average length of a document in D. Assumes all counts are pre-computed in O(|D|Lave) time during one pass through all of the data.
Generally just O(|D|Lave) since usually |C||V| < |
D|Lave
Test Time: O(|C| Lt) where Lt
is the average length of a test document. Very efficient overall, linearly proportional to
the time needed to just read in all the data.
Sec.13.2
Underflow Prevention: using logs
Multiplying lots of probabilities, which are between 0 and 1 by definition, can result in floating-point underflow.
Since log(xy) = log(x) + log(y), it is better to perform all computations by summing logs of probabilities rather than multiplying probabilities.
Class with highest final un-normalized log probability score is still the most probable.
Note that model is now just max of sum of weights…
cNB argmaxcj C
[logP(c j ) log P(x i | c j )ipositions
]
Sec.13.2
Naive Bayes Classifier
Simple interpretation: Each conditional parameter log P(xi|cj) is a weight that indicates how good an indicator xi is for cj.
The prior log P(cj) is a weight that indicates the relative frequency of cj.
The sum is then a measure of how much evidence there is for the document being in the class.
We select the class with the most evidence for it
39
cNB argmaxcj C
[log P(c j ) log P(x i | c j )ipositions
]
Two Naive Bayes Models
Model 1: Multivariate Bernoulli One feature Xw for each word in dictionary
Xw = true in document d if w appears in d Naive Bayes assumption:
Given the document’s topic, appearance of one word in the document tells us nothing about chances that another word appears
This is the model used in the binary independence model in classic probabilistic relevance feedback on hand-classified data.
Two Models Model 2: Multinomial = Class conditional unigram One feature Xi for each word pos in document
feature’s values are all words in dictionary Value of Xi is the word in position i Naïve Bayes assumption:
Given the document’s topic, word in one position in the document tells us nothing about words in other positions
Second assumption: Word appearance does not depend on position
Just have one multinomial feature predicting all words
)|()|( cwXPcwXP ji for all positions i,j, word w, and class c
Multivariate Bernoulli model:
Multinomial model:
Can create a mega-document for topic j by concatenating all documents in this topic
Use frequency of w in mega-document
Parameter estimation
fraction of documents of topic cj
in which word w appears )|(ˆ
jw ctXP
fraction of times in which word w appears among allwords in documents of topic cj
)|(ˆji cwXP
Classification
Multinomial vs Multivariate Bernoulli?
Multinomial model is almost always more effective in text applications
Feature Selection: Why? Text collections have a large number of features 10,000 – 1,000,000 unique words … and more
May make using a particular classifier feasible Some classifiers can’t deal with 100,000 of features
Reduces training time Training time for some methods is quadratic or worse in the number of features
Can improve generalization (performance) Eliminates noise features Avoids overfitting
Sec.13.5
Feature selection: how?
Two ideas: Hypothesis testing statistics:
Are we confident that the value of one categorical variable is associated with the value of another?
Chi-square test (2)
Information theory: How much information does the value of one categorical variable give you about the value of another?
Mutual information
They’re similar, but 2 measures confidence in association, (based on available statistics), while MI measures extent of association (assuming perfect knowledge of probabilities)
Sec.13.5
2 statistic (CHI)
2 is interested in (fo – fe)2/fe summed over all table entries: is the observed number what you’d expect given the marginals?
The null hypothesis is rejected with confidence .999,
since 12.9 > 10.83 (the value for .999 confidence).
)001.(9.129498/)94989500(502/)502500(
75.4/)75.43(25./)25.2(/)(),(22
2222
p
EEOaj
9500
500
(4.75)
(0.25)
(9498)3Class auto
(502)2Class = auto
Term jaguar
Term = jaguar expected: fe
observed: fo
Sec.13.5.2
There is a simpler formula for 2x2 2:
2 statistic
N = A + B + C + D
D = #(¬t, ¬c)
B = #(t,¬c)
C = #(¬t,c)A = #(t,c)
Sec.13.5.2
Feature selection via Mutual Information
In training set, choose k words which best discriminate (give most info on) the categories.
The Mutual Information between a word, class is:
For each word w and each category c
}1,0{ }1,0{ )()(
),(log),(),(
w ce e cw
cwcw epep
eepeepcwI
Sec.13.5.1
Feature selection via MI For each category we build a list of k most discriminating terms.
For example (on 20 Newsgroups): sci.electronics: circuit, voltage, amp, ground, copy, battery, electronics, cooling, …
rec.autos: car, cars, engine, ford, dealer, mustang, oil, collision, autos, tires, toyota, …
Greedy: does not account for correlations between terms
Why?
Sec.13.5.1
Feature Selection Mutual Information
Clear information-theoretic interpretation
May select rare uninformative terms Chi-square
Statistical foundation May select very slightly informative frequent terms that are not very useful for classification
Just use the commonest terms? No particular foundation In practice, this is often 90% as good
Sec.13.5
Greedy inclusion algorithm
Most commonly used in text Algorithm:
• Compute, for each term, a measure of discrimination amongst classes.
• Arrange the terms in decreasing order of this measure.
• Retain a number of the best terms or features for use by the classifier.
• Greedy because • measure of discrimination of a
term is computed independently of other terms
• Over-inclusion: mild effects on accuracy
Feature selection - performance
• Bayesian classifier cannot over fit much
Effect of feature selection on Bayesian classifiers
Naive Bayes vs. other methods
57
Sec.13.6
Benchmarks for accuracy Reuters
•10700 labeled documents •10% documents with multiple class labels
OHSUMED
•348566 abstracts from medical journals
20NG
•18800 labeled USENET postings•20 leaf classes, 5 root level classes
WebKB
•8300 documents in 7 academic categories.