eric c. larson | eclarson.com

flipping the clinic

Assistant Professor Computer Science and Engineering

in home health monitoring using mobile phones

MiPhone 9 Now with iColon.

> slide to unlock

mobileComp Sci. & Engr.

databases & data science

algorithms arch

AI & robotics

OS

networking

languages

symbolic computing

software

mhealth

what is mhealth?

mHealth (em-‘helth )an abbreviation for mobile health, a term used for the practice of medicine and public health supported by mobile devices

the promise of mHealth:eliminate doctor visits remote / automatic diagnosis equalize developing countries

stress check

telemedicinefitness trainer

heart rate

current mhealthremote training43,000 apps for health on the app store

96% are for calorie counting & exercise 4% are remote monitoring !

yet to be a disruptive mHealth technology

consider physician’s needsconnecting with patient tracking baselines personalized trending data managing chronic disease

75% of all US healthcare spending is on chronic disease

source: center for disease control, 2014

mHealth sensing in the home for managing disease

compliance?cost?

privacy?data reliability?

compliance

phone as a sensorbaseline quantitysensor

embedded sensors processing

estimated

accelerometer gyroscope magnetometer /compass dual camera / flash 1+ microphones proximity sensor capacitive sensor gps motorized actuator wireless antenna (s)

compliance++; cost--;

data reliability?

what can the mobile phone sense with clinical accuracy?

ongoing research

lung function jaundice

ongoing research

lung function jaundice

spirometer lung function??

lung function

asthma COPD cystic fibrosis

evaluates pulmonary impairments

spirometer

device that measures amount of air inhaled and

exhaled.

using a spirometer

flow

volume

volum

e

time

using a spirometer

flow

volume

volum

e

time

volume-time graphvo

lume

time

volume-time graphvo

lume

time1 sec.

FEV1

FVCFEV1% = FEV1/FVC

FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity

FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity

FEV1% = FEV1/FVC

> 80% healthy60 - 79% mild40 - 59% moderate

< 40% severe

flow-volume graphflo

w

volume

flow

volumeFEV1 FVC

1 sec.

PEF

PEF: Peak Expiratory Flow FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity

flow-volume graph

flow

volume

normalobstructive

flow-volume graph

obstructive diseases

!

resistance in air path leads to reduced air flow

restrictive diseases

!

lungs are unable to pump enough air and pressure

flow-volume graphFlo

w

Volume

normal

restrictiveobstructive

clinical spirometry

home spirometry

!

faster detection rapid recovery

trending

home spirometry

high cost barrier patient compliance

less coaching limited integration

challenges with

flow rate volume

lung functionairflow sensor

sound pressure microphone processing

estimated

SpiroSmart

availability cost portability more effective coaching interface integrated uploading

Using SpiroSmart

Using SpiroSmart

]

Using SpiroSmart

0 1 2 3 4 5 6 7−1

−0.5

0

0.5

1

time(s)

amplitude

flow features estimation

0 1 2 3 4 5 6 7−1

−0.5

0

0.5

1

time(s)

amplitude

envelope detection

0 1 2 3 4 5 6 7−1

−0.5

0

0.5

1

time(s)

amplitude

resonance tracking0 1 2 3 4 5 6 7

−1

−0.5

0

0.5

1lpc8raw

time(s)

amplitude

flow estimation features

vocal tractsource output 0 1 2 3 4 5 6 7−1

−0.5

0

0.5

1lpc8raw

time(s)

amplitude

auto-regressive estimate

time(s)

frequency(Hz)

1 2 3 4 5 60

500

1000

1500

2000

2500

0 1 2 3 40

5

10

15

Flow

(L/s

)

Volume(L)

0 2 4 6 8 100

1

2

3

4

time(s)

Volu

me(

L)

0 1 2 3 40

5

10

15

Flow

(L/s

)Volume(L)

0 2 4 6 8 100

1

2

3

4

time(s)

Volu

me(

L)

audio

flow features

measures regression

curve regression

lung functionFEV1FVCPEF

study design

x 3

x 3

study enrollment

participants 5218-75 years old, mostly healthy

study a

participants 1012-17 years old, mixed healthy/abnormal

study b

participants 5610-69 years old, mostly abnormal

study c

enrolled by hospitals

resultsmeasures regressionPE

F, Sp

iroSm

art 16

14 12 10 8 6 4 22 4 6 8 10 12 14 16

FEV1

, Spi

roSm

art

!!

6 5 4 3 2 1

1 2 3 4 5 6

FVC,

Spi

roSm

art

!7 6 5 4 3 2

2 3 4 5 6 7 FEV1

%, S

piro

Smar

t100

90 80 70 60 50 40 40 50 60 70 80 90 100

FEV1%, actual

FEV1, actualPEF, actual

FVC, actual

resultsmeasures regressionPE

F, Sp

iroSm

art 16

14 12 10 8 6 4 22 4 6 8 10 12 14 16

FEV1

, Spi

roSm

art

!!

6 5 4 3 2 1

1 2 3 4 5 6

FVC,

Spi

roSm

art

!7 6 5 4 3 2

2 3 4 5 6 7 FEV1

%, S

piro

Smar

t100

90 80 70 60 50 40 40 50 60 70 80 90 100

FEV1%, actual

FEV1, actualPEF, actual

FVC, actual

−2 0 2 4 60

5

10

15

volume(L)

flow(L/s)

−2 0 2 4 60

5

10

15

volume(L)

flow(L/s)

−2 0 2 4 60

5

10

15

volume(L)

flow(L/s)

−2 0 2 4 60

5

10

15

volume(L)

flow(L/s)

−1 0 1 2 3 40

2

4

6

8

volume(L)

flow(L/s)

−1 0 1 2 3 40

2

4

6

8

volume(L)

flow(L/s)

resultscurves regression

can SpiroSmart curves be used for diagnosis?

• 5 pulmonologists

• normal/abnormal subjects curves

• unaware if from SpiroSmart / spirometer

survey

results

normal minimal obstructive mild obstructive moderate obstructive severe obstructive

restrictive

inadequate

error 8%false negative

4%

false positive 14%

one off 10%

identical 64%

!

!

abnormal vs normal

96%

appropriate for trending and screening

global health non-profit

patient and doctors pharmaceutical drug trials

in 10 years, COPD will surpass AIDS/HIV as the leading cause of death in developing nations

ongoing research

lung function jaundice

ongoing research

lung function jaundice

Kernicterus StatisticsKernicterus:+21+($8+mill)+

Hazardous+jaundice:++1158+

($50,000)+

Extreme+jaundice:++2,317+($20,000)+

Severe+jaundice:+35,000+($8,500)+

Phototherapy:+290,000++($1,000)+

Visible+jaundice:+3.5+million+

Births/year:+4.1+million+

In the US Middle- & low-income countries:

• 75,000 cases kernicterus/year

• 114,000 newborn deaths/year

• 65% newborn deaths from kernicterus

kernicterus 21

($8 million)

Bhutani et al., Pediatric Research 2010

Total Serum Bilirubin

Medical GoldStandard

Transcutaneous

TcB

Bilirubinometer

TcBNon-invasive

Correlates 0.75-0.93

$7000

Quick results

Screening tool for TSB

20 !!

15 !!

10 !!

5 !!

0

0.5 1 2 3 4 5 6 Age (days)In Hospital

Biliru

bin

(mg/

dL)

Newborn Bilirubin Levels

75th percentile

25th percentile

Visual Assessment • Parents • Many physicians • Traveling practitioners

In Hospital At Home

Screening Challenges

Tend to underestimate

bilirubin level in blood

jaundice levelblood draw

yellowness camera processing

estimated

bilicam

Study Evaluation100 newborn participants

• <1 day old when enrolled

• 59% white !

2 medical centers !

Data collected by medical professionals using iPhone 4S

Photos: with & without flash

TSB (ground truth)

TcB (control)

3 - 5 days old

Data Collection App

!

Standardize • white balance • card position • phone position

!

Noisy Data

Automatic Quality Control

✔ ✖ ✖

✖ ✖ ✖

Ideal Glare Overexposed

Occlusion Shadow Underexposed

Algorithm Overview

Bilirubin Estimate

Color Balance

Extract Features

Apply Regressions

400 500 600

Wavelength (nm)

Rela

tive

Bili

rubi

n

Abs

orpt

ion

Prob

abili

ty

Bilirubin Absorption Properties

Feature Extraction

Gradient (of RGB channels)

with & without flash

skin

RGB

Cr

CbY

YCbCr

a*

b*

L*

L*a*b*

Extract Features

Apply Regressions

Color Balance

Algorithm Overview

Bilirubin Estimate

Regression EnsembleRegression

Regression

Regression

Regression

RegressionBilirubin Estimate

Combine

Regression Ensemble

No

90th percentileYes

regressions agree

mean

Least Angle Regressions

LARS-Lasso Elastic Net

Support Vector Regressions

k-Nearest Neighbor

Random Forest Regression

Bilirubin Estimate

Results

0

5

10

15

20

25

0 5 10 15 20 25

TSB Ground Truth (mg/dl)

Estim

ated

Bili

rubi

n (m

g/dl

)

Results

0

5

10

15

20

25

0 5 10 15 20 25

TSB Ground Truth (mg/dl)

Estim

ated

Bili

rubi

n (m

g/dl

)

Results

0

5

10

15

20

25

0 5 10 15 20 25

TSB Ground Truth (mg/dl)

Estim

ated

Bili

rubi

n (m

g/dl

) BiliCamrank order 0.85 correlation

Results

0

5

10

15

20

25

0 5 10 15 20 25

TSB Ground Truth (mg/dl)

Estim

ated

Bili

rubi

n (m

g/dl

)

TcBs correlate 0.75 - 0.93

BiliCamrank order 0.85 correlation

TcBrank order 0.92 correlation

Interpretation20 !!

15 !!

10 !!

5 !!

0

high risk

high intermediate risk

low intermediate risk

low risk

Bili

rubi

n (m

g/dL

)

0.5 1 2 3 4 5 6

Age (days)

20 !!

15 !!

10 !!

5 !!

0

high risk

high intermediate risk

low intermediate risk

low risk

Bili

rubi

n (m

g/dL

)

0.5 1 2 3 4 5 6

Age (days)

Bhutani Nomogram

20 !!

15 !!

10 !!

5 !!

0

high riskhigh intermediate risk

low intermediate risk

low risk

Bili

rubi

n (m

g/dL

)

0.5 1 2 3 4 5 6

Age (days)

Interpretation

9 high risk cases based on TSB

20 !!

15 !!

10 !!

5 !!

0

high riskhigh intermediate risk

low intermediate risk

low risk

Bili

rubi

n (m

g/dL

)

0.5 1 2 3 4 5 6

Age (days)

Interpretation

BiliCam 2/9 missed high risk (22%)85% blood draws avoided

20 !!

15 !!

10 !!

5 !!

0

high riskhigh intermediate risk

low intermediate risk

low risk

Bili

rubi

n (m

g/dL

)

0.5 1 2 3 4 5 6

Age (days)

Interpretation

BiliCam 2/9 missed high risk (22%) 85% blood draws avoided

TcB 2/8 missed high risk (25%) 88% blood draws avoided

BiliCam is sufficient for newborn Jaundice screening, but it is unknown how user error affects reliability

ongoing research

lung function jaundice

ongoing research

lung function jaundice

cervical cancer screening

affect sensingintra ocular pressure

oxygen volume VO2

cardiac output & blood pressure

> slide to unlock

Thank You!

eric c. larson | eclarson.com

flipping the clinic

Assistant Professor Computer Science and Engineering

in home health monitoring using mobile phonescollaborators: !Suku Nair Eric Bing Joseph Camp Dinesh Rajan Sohail Rafiqi Mark Wang Shwetak Patel Jim Stout, MD Jim Taylor, MD Margaret Rosenfeld, MD Gaetano Boriello Mayank Goel Lilian DeGreef

eclarson.com eclarson@lyle.smu.edu @ec_larson