alz hack ii
TRANSCRIPT
AlzHack
Data Driven Diagnosis of Alzheimer's Disease
Frank Kelly
Goal definition
Diagnose Alzheimer’s disease as early as possibleBenefit to millions of people (potentially)
Our goal:
Why is Alzheimer’s disease diagnosis important?Chronic neurodegenerative disease
60-70% of dementia cases = Alzheimer's
48 million people affected worldwide (2015)
Wrecks people’s lives (+ their families’)
800,000 people (in the UK) formally diagnosed
Only 43% of those with the condition get a diagnosis
Figures: wikipedia & http://www.bbc.co.uk/science/0/21878238
Demographic changes mean it
will be more widespread
Chart credit: economist.com
By 2050 the number of dementia
sufferers is expected to triple
A global, mounting problem
How is Alzheimer’s disease diagnosed today?
Medical history
Mental status tests
Physical and neurological examination
Blood tests and brain imaging
Example test sheet:: http://www.ftdrg.org/wp-content/uploads/4a-CCT_revised-Picture-stimulus.pdf
A gradual decline
-20 years
-10 years
Death-15 years
-5 years
Earliest Alzheimer’s Mild to moderate Severe
Common diagnosis period
Who are we ?
Full bios: https://alzhack.wordpress.com
What is our approach? We’re doing citizen science● No lab, or lab coats
● Readily available data
● Other people’s research
Diagnose Alzheimer’s disease as early as possibleWhy?
Participate in clinical drug trials Benefit from treatment
More time to plan
Take own decisions
Better carer relationship
Reduce anxieties about unknowns
Sketch: http://www.businessfinancenews.com/28526-will-astrazeneca-plc-and-eli-lilly-give-breakthrough-in-alzheimers/
Design of Study&
Data Collection
How the disease manifests itselfProtein plaques and
tangles accumulate in the
brain:
Disrupting
communication
between nerve cells
Kills nerve cells
Loss of brain tissue
Facts: https://www.alzheimers.org.uk/site/scripts/documents_info.php?documentID=100 Imagery: www.alz.org
How the disease manifests itself (1)
Starts in the hippocampus
Harder to form new memories
Difficult to recollect from days or
hours ago
Video: https://www.youtube.com/watch?v=Eq_Er-tqPsA
How the disease manifests itself (2) ...then takes root in other areas
2. Language processing
3. Logical thought
4. Emotions
5. Senses
6. Older memories
7. Balance and coordination
Video: https://www.youtube.com/watch?v=Eq_Er-tqPsA
Relevant symptoms
Confusion with
time/place
Spatial memory
Problems with words
Misplacing items
Decreased / poor
judgment
Withdrawal from
work
Mood change Difficulty
with familiar
tasks
Challenges in planning
Speech
Short term memory loss
-20 years
-10 years Death
-15 years
-5 years
Earliest Alzheimer’s Mild to moderate Severe
Previously...
Previously: Analysis of a single user’s emails ● An Alzheimer’s
disease sufferer’s
emails over 4 years
● Conversion of email
text to vectors
● Counts, lengths and
other metrics
FeaturesMemory, language and sentiment related metrics extracted
Results
Some “explainable” trends
ChallengesSingle user: lack of data and likely bias
Scaling up: security concerns & deletion
How did we get more data?
Forum post scraping
First lxml, then BeautifulSoup
● Two sub-forums
● ~3,600 threads
● ~78,000 posts
○ Post content
○ Post metadata
○ User metadata
Data preparation
Content punctuation
sanitised by regexp
substitutions.
Sub forum
post data
(x2)
User labelling
How do we label a user?● Users frequently post in both sub-forums
● To differentiate:
○ Assume that OPs (thread starters) in a sub-forum are of that category
○ Otherwise look at ratio of posts (replies) between the two sub forums*
FP = First Post in thread SP = Subsequent Post in thread
How do we label a user?
Thread
Reply
Dementia Partner
Discard Unknown
Features and EDA
Sentiment “polarity”
(out-of-the-box via
NLTK & TextBlob)
● Alternatively can
train your own text
classifier:
http://streamhacker.com/2010/05/10/text-classification-sentiment-analysis-naive-bayes-classifier/
‘Mood change’as a feature
● Average of sentence
sentiments per post
● Slightly higher
sentiment for
dementia sufferers’
posts
Language-oriented featuresLexical functions
Comprehension functions
Empty phrases
Paraphasias and
neologisms
Vocabulary-related
Readability
“Go ahead” phrases
Unintended or
invented words
Difficult words count
Dale-Chall readability
Flesch Kincaid
Flesch Reading Ease
Counts of “ummm...errr”
Words that are not in
common usage
Simple language features● Sentence count
● Word count
● Words per sentence
● Unique word count
● Unique words to total ratio
● “Go Ahead” words (Empty phrases)
Readability(package readability-lxml)
● Avg syllables per word
● Avg letter per word
● Flesch reading ease
● Flesch kincaid grade
● Polysyllabcount
● Automated readability index
● Number of “difficult” words
● Dale-chall readability score
● Gunning fog
Vocabulary & word counts
Memory-oriented features● Sort posts by username and timestamp, add a shifted column
Apply comparison function
between post and previous post:
○ NLTK edit_distance (fuzzy
match)
○ Cosine similarity between TF-
IDF vectors
Part of speech (POS) features● Tag words and
tally up
frequencies
● Calculate
“rates”
Models & results
Explanatory or predictive modelling ?
● Actually both.
● First ‘interpret’ a classifier (explanatory)
● Secondly need a ‘real-time’ detection system (predictive)
Data modelling strategy (used for initial ML runs)Aggregation of posts
● pandas: groupby, agg by username
Balancing out the dataset
● Many more partner users than sufferers
● Subsample larger (partner) dataset to even things up
Validate using random train and test sets
● Randomly select 80% of users for training, 20% test
Model Results for Misc. Features● Median values (aggregated over all posts per user)
Best: SVM Radial basis function classifier (with grid
search)
User classification accuracy: 57%
Model Results for Memory Features● Median values (aggregated over all posts per user)
Best: K-nearest neighbours Classifier
User classification accuracy: 63%
Model Results for Readability Features● Median values (aggregated over all posts / user)
Best: K-nearest neighbours Classifier
User classification accuracy: 59%
Model Results for Part-Of-Speech Features● Median values (aggregated over all posts per user)
Best: SVM Radial basis function classifier (with grid
search)
User classification accuracy: 61%
Model Results for All Features● Median values (aggregated over all posts per user)
Best: Naïve Bayes Classifier
User classification accuracy: 63%
Re-think: Classify posts, not users
● Currently group by userID
● Some users post more than others
● Posts would utilise full “richness” of the dataset
● Double round of sampling required on post set:
○ 3 - 4 times more “partners” than dementia sufferers
○ Partners post approx. 3 times more posts than sufferers do
Model Results for All Features (by post)● Filtered set of posts
Best: Random Forest Classifier
Accuracy of 68% percent in ability to classify a post
Wrap up
Results in summary● Best performing feature group so far on aggregated set by user:
○ Memory-based features
● Best performing individual feature on aggregated set by user:
○ Verb rate = ratio of verbs to word count in post
● Best performing individual feature on individual post:
○ Cosine similarity to previous post
● Aligns with symptoms expected in early stage to mild dementia
Future avenues● Data
○ Further data gathering (more blogs including non-alzheimer's topic blogs)
○ Better user identification (e.g. active learning)
● Features
○ More and better
○ Types of individual dementia distinguish
○ More memory-related features (e.g. LSI)
● Clustering of posts into ‘topics’ or users into ‘types’
○ gensim / LDA topic modelling
○ Early stage / medium condition / advanced condition posters
● Classification and modelling
○ Time series analysis
○ New sampling techniques, input validation and models
Future: Time series analysis
● Noisy datasets
○ Apply numerical Bayesian
inference
● Are we looking for a steady
change in the mean?
○ Ramp detection
● Or a sudden change in
variance?
○ Step change detection
Dementia sufferer
Partner
Conclusions● Introduction to Alzheimer’s and its impact
● Explanation of our technical approach and surrounding challenges
● Initial observations and predictions
● Tough problem and a worthwhile cause for data science
● Please contact us if you would like to help, or have ideas:
[email protected] https://alzhack.wordpress.com/contribute-2/
Thank you!