Slide 1

KAIST

Slide 2

Smart Phone/Pad Sensor Applications Resizing screen/tilt Environment adjustment of apps for users comfort Adjustment in cinema, movement prediction Gaming Augmented Reality (AR) AR Gaming AR Navigation Bar codes Geotagging, recommendations.. Network of objects, locations and people, 3D social networking? Distributed sensor systems Noise mapping, traffic information, etc. And anything you can imagine

Slide 3

Vehicular Sensing People Sensing

Slide 4

I. KAIST Slides based on: Sensor Fusion on Android Devices A Revolution in Motion Processing, David Sachs, Feb. 2010 Android Sensors, Stefan Varga, Michal Kostic, Oct. 2010

Slide 5

Contents Elements of mobile sensor networking Programming Android sensor/location service Setting up sensor/location service Processing events How do sensors work? Accelerometer, Magnetometer, Gyroscope, Light Sensor, Proximity Sensor, GPS, Microphone, etc. How do location services work? GPS, cell-tower location, Wi-Fi positioning

Slide 6

Elements of Mobile Sensor Networking network infrastructure Mobile sensors r vehicles, smartphones r run sensing applications r may be stationary (non- mobile) or mobile

Slide 7

Elements of Mobile Sensor Networking network infrastructure Wireless nets r mobile to base station r base station: e.g., cell towers, 802.11 access points; connected to a wired network r (also, mobile-to-mobile: Bluetooth, Wi-Fi ad hoc) r multiple access protocol coordinates link access r various data rates, transmission distance Bluetooth, WiFi Ad Hoc Wi-Fi, 3G/LTE/WiMax

Slide 8

Elements of Mobile Sensor Networking network infrastructure Computing infra r back-end platform for data processing and web services r could be part of network infra (or independent) r centralized or de- centralized infra

Slide 9

Wireless Networking Standards Indoor 10-30m Outdoor 50-200m Mid-range outdoor 200m 4 Km Long-range outdoor 5Km 20 Km.056.384 1 4 5-11 54 GSM, IS-95, CDMA 2G UMTS/WCDMA, CDMA2000 3G 802.15 802.11b 802.11a,g UMTS/WCDMA-HSPDA, CDMA2000-1xEVDO 3G cellular enhanced LTE, 802.16 (WiMAX) 200 802.11n Data rate (Mbps) All IP Wi-Fi Bluetooth 4G

Slide 10

Contents Elements of mobile sensor networking Programming Android sensor/location service Setting up sensor/location service Processing events How do sensors work? Accelerometer, Magnetometer, Gyroscope, Light Sensor, Proximity Sensor, GPS, Microphone, etc. How do location services work? GPS, cell-tower location, Wi-Fi positioning

Slide 11

Smart Phone/Pad Sensors Nexus OneNexus SiPhone4 Samsung Galaxy S HTC Incredible Galaxy Tab/ iPad2 Accelerometer OOOOOO Magnetometer OOOOOO Gyroscope OO?O Light OOOOOO Proximity OOOOOO Camera OOOOOO Voice OOOOOO GPS OOOOOO

Slide 12

Android System Architecture

Slide 13

Android API (I) Package: android.hardware Classes: SensorManager Android sensor service Sensor specific sensor SensorEventListener SensorEvent specific event of the sensor = data LocationManager Android location service LocationProvider (e.g., WPS, Network, GPS) LocationListener Processing updates for location (Location) or provider (LocationProvider)

Slide 14

Android API Most sensors interfaced through SensorManager or LocationManager Obtain pointer to android service using Context.getSystemService(name) For name, use constant defined by Context class SENSOR_SERVICE for SensorManager LOCATION_SERVICE for LocationManager Check for available sensors using List mgr.getSensorList(int type) Type constants provided in Sensor class documentation Check for available location providers using List providers = mgr.getAllProviders();

Slide 15

SensorManager Use getDefaultSensor(int type) to get a pointer to the default sensor for a particular type Sensor accel = sensorManager.getDefaultSensor( Sensor.TYPE_ACCELEROMETER); Register for updates of sensor values using registerListener(SensorEventListener, Sensor, rate) Rate is an int, using one of the following 4 constants SENSOR_DELAY_NORMAL (delay: 200ms) SENSOR_DELAY_UI (delay: 60ms) SENSOR_DELAY_GAME (delay: 20ms) SENSOR_DELAY_FASTEST (delay: 0ms) Use the lowest rate necessary to reduce power usage Registration will power up sensor: mSensorService.enableSensor(l, name, handle, delay);

Slide 16

SensorManager Unregister for sensor events using unregisterListener(SensorEventListener, Sensor) or unregisterListener(SensorEventListener) Undregistering will power down sensors: mSensorService.enableSensor(l, name, handle, SENSOR_DISABLE) Perform register in OnResume() and unregister in OnPause() to prevent using resources while your activity is not visible SensorListener is deprecated, use SensorEventListener instead See documentation for Sensor, SensorManager, SensorEvent and SensorEventListener

Slide 17

SensorEventListener Must implement two methods onAccuracyChanged(Sensor sensor, int accuracy) onSensorChanged(SensorEvent event) SensorEvent int accuracy Sensor sensor long timestamp Time in nanoseconds at which event happened float[] values Length and content of values depends on sensor type

Slide 18

API Setup public class MainActivity extends Activity implements SensorEventListener {.. private SensorManager sm = null; public void onCreate (Bundle savedInstanceState) {.. sm = (SensorManager) getSystemService(SENSOR_SERVICE); } protected void onResume () {.. List typedSensors = sm.getSensorList(Sensor.TYPE_LIGHT); // also: TYPE_ALL if (typedSensors == null || typedSensors.size() Westwood) Manhattan MobilityReal-track Mobility Time (s) Fraction of actively harvested meta-data">

Meta-data Harvesting Results Meta-data harvesting: agent actively harvests meta-data Impact of node density (#nodes), speed, mobility Higher speed, faster harvesting Higher density, faster harvesting (more # of meta-data from neighbors) Less restricted mobility, faster harvesting (Man>Westwood) Manhattan MobilityReal-track Mobility Time (s) Fraction of actively harvested meta-data

Slide 65

Trajectory Tracking Agent re-constructs the trajectory of a node

Slide 66

CarTel: A Distributed Mobile Sensor Computing System Bret Hull, Vladimir Bychkovsky, Kevin Chen, Michel Goraczko, Allen Miu, Eugene Shih, Yang Zhang, Hari Balakrishnan, and Samuel Madden Sensys 2006

Slide 67

CarTel System Architecture Open wireless Access Point Users wireless Access Point Adapters log GPS, OBD, camera data Data sent via prioritized continuous queries ICEDB Remote Delay-tolerant relay via WiFi CafNet PortalClients Prioritizes data Answers local snapshot queries Logs continuous query results ICEDB Server

Slide 68

CarTel S/W Architecture Portal Data Visualization CafNet Stack CQ Web Server ICEDB Server OBD-II WiFi Monitor Camera Traffic Speed/ Delay Portal Applications Portal Streaming Sensor Data Cont. queries + adaptors

Slide 69

Intermittently connected DB (ICEDB) ICEDB server Maintains a list of continuous queries submitted by applications Queries are pushed to mobile nodes using CafNets Results from ICEDB clients are stored in the RDBMS at the portal ICEDB client Process the sensed data and return the query results using CafNet (carry-and-forward net) Prioritize the result streams in the order of importance

Slide 70

Intermittently connected DB (ICEDB) Queries in ICEDB are written in SQL with several extensions for continuous queries and prioritization EX)SELECT carid, traceid, time, location FROM gps WHERE gps.time BETWEEN now()-1 mins AND now() RATE 5 mins Prioritization is required since delivering data in FIFO order is suboptimal in bandwidth- constrained network Intermittent connectivity due to high speed mobility and restricted mobility patterns

Slide 71

Portal Users navigate sensor data in CarTel using a web- based interface Main components of the portal Portal framework ICEDB server to retrieve sensor data Data visualization library to display geo-coded data Trace: all sensor data collected during a single trip (i.e., between ignition on and off)

Slide 72

The Pothole Patrol: Using a Mobile Sensor Network for Road Surface Monitoring Jakob Eriksson, Lewis Girod, Bret Hull, Ryan Newton, Samuel Madden, Hari Balakrishnan MIT Computer Science and Artificial Intelligence Laboratory Mobisys 2008

Slide 73

Mobile Road Surface Monitoring Potholes: hazardous to drivers and increasing repair costs due to vehicle damage Determine which roads need to be fixed Static sensors will not do well requires mobility! Challenges: Differentiate potholes from other road anomalies (railroad crossings, expansion joints) Coping with variations in detecting the same pothole (speed, sensor orientation)

Slide 74

Pothole Patrol: Architecture Pothole Record Clustering Cab 1 GPS 3 Axis Accelerometer Location Interpolator Pothole Detector Cab 2 GPS 3 Axis Accelerometer Location Interpolator Pothole Detector Central Server

Slide 75

Sample Traces Smooth RoadRail Crossing PotholeExpansion Joint

Slide 76

Detection Algorithm Features of accelerometer data High energy events are potholes? Not really! Rail road crossings, expansion joints, door slamming are high energy events Accelerometer data is processed by embedded computer 256-sample windows Pass through 5 different filters

Slide 77

Detection Algorithm Speed Car is not moving or moving slowly Rejects door slam and curb ramp events IN Windows of all event classes SpeedHigh-passz-peak xz-ratio speed vs. z ratio OUT Pothole Detections

Slide 78

Detection Algorithm High-Pass Removes low-freq components in X- and Z-axes Filters out events like turning, veering, braking. IN Windows of all event classes SpeedHigh-passz-peak xz-ratio speed vs. z ratio OUT Pothole Detections

Slide 79

Detection Algorithm z-peak Prime characteristic for minor anomalies Rejects all windows with absolute z-acceleration < t z IN Windows of all event classes SpeedHigh-passz-peak xz-ratio speed vs. z ratio OUT Pothole Detections tztz

Slide 80

Detection Algorithm xz- ratio Assumes potholes impact only side of the vehicle Identifies anomalies that span width of the road (rail crossings, speed bumps) high z-peak + low x-peak vs. high z-peak + high x-peak (potholes) x peak (within w (=32) samples from z peak ) < t x * z peak IN Windows of all event classes SpeedHigh-passz-peak xz-ratio speed vs. z ratio OUT Pothole Detections tztz txtx

Slide 81

Detection Algorithm speed vs. z ratio At high speeds, small anomalies cause high peak accelerations Rejects windows where Z peak < t s X speed IN Windows of all event classes SpeedHigh-passz-peak xz-ratio speed vs. z ratio OUT Pothole Detections tztz txtx tsts

Slide 82

Detection Algorithm Filtering parameter threshold tuning via training Thresholds: z-peak, xz-ratio, and speed vs. z ratio Perform exhaustive search over a range of values Improving detection accuracy with clustering Cluster of at least k events must happen in the same location with small margin of error(d) Clustering algorithm Place each detection in d X d grid. Compute pair-wise distances in same or neighboring grid cells Iteratively merge pairs of distances in order of distance Max intra cluster distance < t Reported location is the centroid of the locations

Slide 83

Performance on labeled data Randomly divided into training set and test set Cf: loosely labeled data (extra information about manholes, bridges, etc.) Performance ClassHand Labeledw/ Loosely Labeled Pothole88.9%92.4% Manhole0.3%0.0% Expansion joints2.7%0.3% Railroad Crossing8.1%7.3%

Slide 84

Contents Vehicular sensing MobEyes CarTel Pothole patrol People sensing PIER SoundSense

Slide 85

PIER: Personal Environmental Impact Report, as a Platform for Participatory Sensing Systems Research Min Mun, Sasank Reddy, Katie Shilton, Nathan Yau, Jeff Burke, Deborah Estrin, Mark Hansen, Eric Howard, Ruth West, Peter Boda UCLA, Nokia Mobisys 2009

Slide 86

Participatory Sensing Distributed data collection and analysis at the personal, urban, and global scale Individuals and communities make decisions about when and how to Capture Store Access Analyze Share Participants use mobile phones to gather data and web services to aggregate and interpret the assembled information.

Slide 87

Health and Wellness: Personal Environmental Impact Report CO2 emissionsFast food exposure PM2.5

Slide 88

Personal Environmental Impact Report Carbon Impact: a measure of transportation- related carbon footprint Sensitive Site Impact: a users transportation related airborne particulate matter emissions near sites with populations sensitive to it Smog Exposure: a users transportation-related exposure to particulate matter emissions Fast Food Exposure: the time integral of proximity to fast-food eating establishment Ultimately we want people to take notice of impact and exposure and be able to start conversations. The absolute numbers are not whats key, but instead, trends over time. How can one reduce impact and minimize exposure?

Slide 89

PEIR as a Participatory Sensing System "Sensing Pollution without Pollution-Sensors" Existing Infrastructure Annotation /Inferences Scientific Models Activity Classification e.g., staying, walking, driving GIS Data Annotation e.g. weather, traffic Impact and Exposure Calculation Data Aggregation Tracklog format School,hospital,fast food restaurant locations Weather, traffic data User profile

Slide 90

PEIR as a Participatory Sensing System

Slide 91

Slide 92

SoundSense: Scalable Sound Sensing for People-Centric Applications on Mobile Phones Hong Lu, Wei Pan, Nicholas D. Lane, Tanzeem Choudhury, Andrew T. Campbell Dept. of Computer Science, Dartmouth College Mobisys 2009

Slide 93

SoundSense allows us to sense people and the environment

Slide 94

SoundSense Architecture Admission Control Acoustic Features Decision Tree Classifier Markov Model Recognizer Ambient Sound Learning Voice Analysis Music Analysis Sound Waveform Framing & Admission Control Feature Extraction Coarse Category Classification Intra-Category Classification

Slide 95

Admission Control Filter out silence and non-informative frames Criteria: Energy & Spectral Entropy

Slide 96

Discriminative Acoustic Features Frequency Spectrum based features: Robust to distance/muffling Computationally very cheap after initial FFT

Slide 97

Discriminative Acoustic Features Spectral Rolloff for Music and Voice Spectral Rolloff Time

Slide 98

Coarse Category Classification Acoustic Features Decision Tree Classifier Ambient Sound Voice Music S1S2 S3 S1S2 S3 S1S2 S3 Markov Model Recognizer Music, Music, Voice, Music, Voice, Music, Music Music, Music, Music, Music, Music, Music, Music Confusion matrix for the decision tree classifier with Markov model smoothing

Slide 99

Learning Significant Sound Unique to the individual Not practical to build supervised classifiers for all the potential significant sounds Unsupervised learning! Examples: Mystery Sound 1 Car (turning signal) Mystery Sound 2 Vacuum Cleaner Incrementally build models of encountered sounds Rank them by order of importance Human-in-the-loop labeling

Slide 100

Evaluation Friday Saturday

Slide 101

: Privacy by Design - Principles of Privacy-Aware Ubiquitous Systems, Marc Langheinrich, Ubicomp 2001 Location Disclosure to Social Relations: Why, When, & What People Want to Share, Consolvo et al., CHI'05 duty-cycling trade-offs (WiFi vs. 3G) : Decomposing power measurements for mobile devices, Andrew Rice and Simon Hay, Percom 2010 Coolspots: Reducing the power consumption of wireless mobile devices with multiple radio interfaces, Pering et al., Mobisys 2006 Activity classification activity classification : Activity Recognition from User-Annotated Acceleration Data, Pervasive 2004

Slide 102

network infra , , Vehicular Sensing People Sensing

Slide 103

MobiSys: http://www.sigmobile.org/mobisyshttp://www.sigmobile.org/mobisys Ubicomp: http://www.ubicomp.orghttp://www.ubicomp.org Pervasive: http://pervasiveconference.orghttp://pervasiveconference.org PerCom: http://www.percom.org/http://www.percom.org/ SenSys: http://sensys.acm.orghttp://sensys.acm.org MobiCom: http://www.sigmobile.org/mobicomhttp://www.sigmobile.org/mobicom