Efficient Similarity Search on Big Datawith office laptopSergii ShelpukHead of Data Science, V.I.Tech

The ProblemYou have a database of 30M patients with all medical records. Each patient described by 250K of binary features.

You need a system for finding N most similar patients to a given one.Jesus Christ, its Big Data, get Hadoop!

Jesus Christ, its Big Data, get Hadoop!

Can we do better?Two main ideas:we dont need the meaning of each feature, we only care about similarity of the patients;we dont want to compare very different patients, we want to compare only the most similar ones.

Step 1: Reduce dimensionalityDecrease dimensionality of the data while preserving similaritiesLocality-sensitive hashing and minhashing

K-Means clusteringK-Means clustering groups similar patients in one group

Step 2: Group similarGroup similar patients and store groups as separate filesStore centroids of each cluster in a separate file, too

ApproachTo find N similar patients:Load a patientReduce dimensionality with minhashingLoad centroid fileCompare patient to every centroidLoad cluster file of the closest centroidCompare patient with patients in the clusterShow top N similar

Results50000 clusters up to ~1000 patients per cluster~500Kb-1Mb of every cluster file~18Mb centroid file

To do similarity search you need:~20Gb HDD~20Mb RAMSearch works in ~100 milliseconds on a regular office laptop

Thank you?