weather underground - pws, data ingestion and apis

Hello Sensor

1

Agenda1. Weather Underground Introduction 2. Making Your Own PWS 3. Data Ingestion & QC 4. API

2

Weather Underground Intro

3

What is Weather Underground?● Web● Flagship app● Storm● WunderStation● PWS Network● API

4

Web

● Powered by 200k+weather stations

● Visually engaging● Provides low-level

weather data

5

Flagship App

● The most hyperlocal forecasts● Data presented in a stunningly

simple interface

6

Storm

● The best app for the worstweather

● Highest resolution radar● Severe weather alerts

7

WunderStation

● Personalized weather dashboard

● Features your own PWS data

8

PWS Network● There are about 12k

government provided weather stations

● We fill in the gaps withover 200k PersonalWeather Stations

9

Making Your Own PWS

10

What is a Weather StationTraditional stationsQualitative reporting (crowd reports)Image recognitionPhone SensorsCar sensorsMaker Station

11

Weather hungry data monstersTo serve globally we need more data-Engage with local met offices (if they exist)-Engage with education/maker community

More data, better data = better forecasts.

12

Roll your ownOpen source weather stations make IoT and weather more available/flexible for local needs

Can be part of an education program

13

What does it take1.Sensor (Temp, precip, humidity, uv, etc)2.Controller (arduino, particle, etc) 3.Memory and/or Transmitter (flash,wifi, cellular)4.Power (solar, battery, mains)

14

Station challengesHardware:1:power (limits everything)2:transmit (expensive power budget item)3:durability (usually moving parts)

4:sensors (minor technical issues)5:controller (very low requirements)

15

Station challengesBiggest contributor to data variation: Enclosure design

The Effectiveness of the ASOS, MMTS, Gill, and CRS Air Temperature Radiation Shields: K. G. Hubbard, X. Lin, and E. A. Walter-Shea 16

Tiny wifiTiny wifi connected station

limited battery life

Used to monitor terrarium

17

Ol faithfulGood reliability, online over a yearSolar and battery powered

Enclosure made from ~$6 garden supplies

Particle Photon (WiFi)

Spark Fun Weather Shield -HTU21D humidity sensor-MPL3115A2 pressure sensor

18

Cell-o thereParticle Electron: cell radio + microcontroller BMP280: temp, humidity, pressure sensorEnclosure made from a painted soda cup

Data is good if kept in shade however: no venting = heat buildupok proof of concept, needs refinement

19

Data Ingestion & QC

20

Ingestion

Rapidfire● Ingests and stores data reported at rates as fast as one observation

every 2 seconds● Stores data in current condition file, records history data at as high

resolution as once every 5 seconds

21

Quality Control (QC)Before QC

22

Quality Control (QC)After QC

23

Quality Control (QC)

24

The QC Checks

● Range Check● Stuck Sensor Check● Neighbor Check

25

Range CheckHave these readings ever happened on Earth?

Temperature < -130º F or > 135º F.Dew Point < -90º F or > 90º F.

Wind Speed < 0 mph or > 279 mph. Wind Direction < 0º or > 360º.

Pressure < 846 inHg or > 1100 inHg.

26

Stuck Sensor CheckHas the temperature changed in the past 6 hours?

● by at least 0.1°F● lack of change is often an indication of

other stuck sensors as well

27

Neighbor CheckIs the temperature of this station similar to the majority of stations nearby?

● collect sensors in 15 km of current sensor● find clusters divided by 3° F● determine majority cluster(s)● throw out statistical outliers

Most essential customer-facing check

28

Neighbor Check

29

The Next Step - QC on Ingest● Current QC

○ cycle is 15 minutes, allowing bad observations to linger on the site and apps during that time

○ written in difficult to maintain and extend multi-threaded C++ code

● IBM Streams + QC○ clean obs all the time○ written in single threaded Python with better performance, stability,

extensibility, third-party libraries like Spark, and support for modern technologies like JSON and REST

30

API

31

200,000+ Personal Weather Stations

2.2 Billion forecast locations | 180 M consumers / month 32

33

Uptime: 99.95 % Latency ~25 ms

Autoscale to 20B requests per day

Scalability Average 10s of Billions requests per day

Global Coverage (US East, US West, EU, Asia)

Partial DeploymentsVersioned artifacts and rollbacks

Faster code to prod: Less dependency b/w teams

Your favorite tech / language here

34

Architecture: Storage Polyglot

Real time data and caching

Historical weather data Data Migration

Gateway Data Analytics

ArchivesImagesVideos

AnalyticsInformatica

Drupal

35

Thank you!

36

Questions?

37