Independent Component Analysis

From PCA to ICABell Sejnowski algorithmKurtosis methodDemonstrations

Bell and Sejnowski 1995

Consider y=g(x)+noise with f depending on w

I(y;x)=H(y)- H(y|x)

H(y|x)=E_x E_y|x [-log P(y|x)]

ICA based on KurtosisOja and Hyvarinen

Independent Component Analysis

Perform “blind separation” of signals recorded at multiple sensors

Use minimal assumptions about the characteristics of the signal sources.

An overview of applications of ICA to biological data and general data mining,Computational Neurobiology Laboratory Salk Institute, La Jolla CA (April, 1999).

Enter [Enter] to advance, [up-arrow] to rewind.

Principle: Maximize Information

• Q:Q: How to extract maximum

information from multiple visual

channels?

Set of 144 ICA filters

• AA: ICA does this -- it maximizes

joint entropy & minimizes

mutual information between output

channels (Bell & Sejnowski, 1995).• ICA produces brain-like visual filters

for natural images.

ICA versus PCA

• Independent Component Analysis (ICA) finds directions of maximal independence in non-Gaussian data (higher-order statistics).

• Principal Component Analysis (PCA) finds directions of maximal variance in Gaussian data (second-order statistics).

Example: Audio decomposition

Play Mixtures Play Components

Perform ICA

Mic 1

Mic 2

Mic 3

Mic 4

Terry Scott

Te-Won Tzyy-Ping

Electroencephalography (EEG)

• ICA separates

brain signals from

artifacts.

Artifacts

Brainsignals

• Allows monitoring

of multiple brain

processes.

• Permits study of

brain activity in

noisy conditions.

Functional Brain Imaging

• Functional magnetic

resonance imaging (fMRI)

data are noisy and

complex.

I C A C o m p o n e n t T y p e s

S u s t a i n e d t a s k - r e l a t e d

( a )

T r a n s i e n t l yt a s k - r e l a t e d

( b )

S l o w l y - v a r y i n g

( c )

Q u a s i - p e r i o d i c

( d )

A b r u p t h e a dm o v e m e n t

( e )

A c t i v a t e dS u p p r e s s e d

S l o w h e a dm o v e m e n t

( f )

• ICA identifies concurrent

hemodynamic processes.

• Does not require a priori

knowledge of time courses

or spatial distributions.

Data Mining

• ICA was applied to Armed Forces Vocation Aptitude Battery (ASVAB) test scores and Navy Fire Control School grades.

• ICA may suggest more efficient and balanced selection criteria.

• Two ICA components contributed to final school

grade.

This presentation by

• Scott Makeig, Naval Health Research Center, San Diego

• Tzyy-Ping Jung, Institute for Neural Computation,

UCSD, La Jolla CA

• Te-Won Lee, Salk Institute, La Jolla CA

• Sigurd Enghoff, Salk Institute

• Terrence J. Sejnowski, Salk Institute & UCSD

From Barak Pearlmutter Contextual ICA The first demo applies the Contextual ICA blind source

separation algorithm. Lucas Parra and I digitally extracted ten five-second clips from ten audio CDs. These were digitally mixed, without time delays or echos, and with random gains, to form the output of a synthetic microphone. Ten such microphone outputs were synthesized. These synthetic microphone outputs formed the input to the Bell-Sejnowski Independent Components Analysis algorithm. The sources are somewhat separated in the output of the Bell-Sejnowski ICA algorithm, but not fully.

The same synthetic microphone outputs were then used as input to our new cICA algorithm (see my publications page for technical details). The sources are almost fully separated in the output of cICA.