teradata partner 2016 gas_turbine_sensor_data
TRANSCRIPT
#TDPARTNERS16 GEORGIA WORLD CONGRESS CENTER
Gas Turbine Sensor Data-Plotting and Alarm Visualization Dashboard
Bruce Baum, Enterprise Architect, Siemens
Jose M. Borja, TDWH Architect, Bormar Consulting LLC
• Siemens and Power Generation Services
• Business Problem
• Benefits of The Approach
• Solution Data Architecture
• Solution Technical Highlights
Session Agenda
2
• Siemens and Power Generation Services
• Business Problem
• Benefits of The Approach
• Solution Data Architecture
• Solution Technical Highlights
Business Perspective
3
My Background and Role
4
• Career as Solution Architect, Technology Strategist, and Developer
• Joined Siemens in 2003
• Current Role as Big Data Architect
• Primary focus on Database Technologies
• Extensive knowledge of data used by Power Generation Services
• Leveraging Teradata and its Hadoop Appliance
• 100’s of Billions of Sensor Readings on Hadoop
• Integrating sensor data with master and transactional data
• Data modeling to deliver semantic views
Siemens – Who we are
5
Electrification, Automation and Digitalization are long-term growth fields of Siemens.
Power and Gas Wind Power and Renewables Power Generation Services
Energy Management Building Technologies Mobility
Digital Factory Process Industries and Drives Healthcare
Shifting markets drive need for answers
6
Predictive Maintenance key to Service
7
Optimizing Gas Turbine operations
8
Our solution will highlight how we organize and process relevant sensor properties in order to quickly identify trends or events that contribute to conditions such as elevated NOx emissions.
Power Generation Services
9
Business Problem
• Engineers needed to monitor operating conditions of large-scale gas turbines for an extended period of time to detect abnormal trends
• Using readily-available tools (Access, Excel), very large amounts of sensor data were collected and cleansed as input to visualization tools
• Due to tool limitations, only two months of data could be analyzed
• One day each week was needed to prepare this limited data set
• This technique would not scale for a larger number of turbines
Impact and Value to Business
10
Benefits Realized
Business no longer has to repeatedly acquire and organize data
Replaced by automated, daily loading of machine data collected from turbines worldwide
No longer limited to a handful of turbines or analyzing for only a short-time period.
Any turbine since its beginning of operating life can be analyzed (10+ years)
Billions of rows of “raw” sensor data can be efficiently analyzed
Impact and Value to Business
11
Unit A Unit C
Unit A
Unit B
Unit C
Weekly Flashback Events
Visualizations proved helpful to demonstrate a correlation between combustor “flashback” events and elevated NOx levels.
• Siemens and Power Generation Services
• Business Problem
• Benefits of The Approach
• Solution Data Architecture
• Solution Technical Highlights
Technical Perspective
12
“Big” Data Architecture
13
Raw Sensor Data
Schema On Read Sensor
Data
Hadoop with Hive
Integrated Core Sensor Data
Dimensional
Teradata DWH
Mart
Unified Data Architecture
QueryGrid
Source Sensor Data
VolumeDetailSparse
AnalyticallyEnriched,
Non-Sparse
300B+ rows
75B+ rows
30M+ rows
100% BTEQ ELTL
“Big” Data Architecture
14
Core Sensor Data
Dimensional
Mart
Mart is a Semantic Layer built on top of Core and BI• Access Dimensional Layer (Facts and Dimensions)• Access Core Layer (Detail Sensor Data)• Semantic Views delivering “Smooth” Time Series Data from Core• Filters Dimension Data - Applies Mart/Department specific rules• Security managed at mart level
Tableau
Ad-Hoc Consumers
Hive Sensor DataHive too slow but it is nice to have as an option
“Big” Data Architecture
15
Integrated Core in 3NFGas Turbine
Gas Turbine Id
Descriptors
Gas Turbine Sensor Measure
Measure Timestamp
Gas Turbine Id (FK)
Sensor Id (FK)
Measure Value
Sensor
Sensor Id
Descriptors
FACT Gas Turbine Daily
Gas Turbine Id (FK)
Sensor Id (FK)
The Date (FK)
The Time (FK)
Sensor A Value
Sensor B Value
....
Sensor Y Value
Sensor Z Value
DIM Date
The Date
Descriptors
DIM Time
The Time
Descriptors
DIM Sensor
Sensor Id
Descriptors
DIM Gas Turbine
Gas Turbine Id
Descriptors
Dimensional Model
• 75B+ rows, sparsely populated• Time Series not aligned• Good for plotting• Difficult for users to do analytics• PPI on all PKs
• 1 Minute Grain @ Turbine level ~30M rows• Denormalized – 1 sensor measure per column• Multiple Fact tables by area of interest• Multi Table Join Indices implement semantic views• Enhanced with Analytical Sensor data (derived sensors)
BI Enrichment
“Big” Data Architecture
16
Integrated Core in 3NF
Gas Turbine
Gas Turbine Id
Descriptors
Gas Turbine Sensor Measure
Measure Timestamp
Gas Turbine Id (FK)
Sensor Id (FK)
Measure Value
Sensor
Sensor Id
Descriptors
Dimensional Model
• Analytical events ~ 1M rows• Complex calculations of interest
BI Enrichment
FACT Gas Turbine Event
Gas Turbine Id (FK)
Sensor 1 Id (FK)
Sensor 2 Id (FK)
The Date (FK)
The Time (FK)
Sensor 3 Id (FK)
Event Type (FK)
Event Period
Min Sensor 1 Value
Max Sensor 1 Value
.......
Event Duartion Seconds
DIM Date
The Date
Descriptors
DIM Time
The Time
Descriptors
DIM Sensor
Sensor Id
Descriptors
DIM Event Type
Event Type
Descriptors
DIM Gas Turbine
Gas Turbine Id
Descriptors
• 75B+ rows, sparsely populated• Time Series not aligned• Good for plotting• Difficult for users to do analytics• PPI on all PKs
ELTL UDA Performance
17
BTEQ scriptCalls Stored Procedure
Builds Dynamic SQLInserts into TD STAGESELECT FROM Foreign
Server (Hive)
Sensor Data Challenges
18
• So simple, 3 dimensions, yet so complex
• Sensor data needs context to be of use
• Sensor data can be fast or slow
• Sensor data often captured with “Dead bands” (reading when value changes specific %)
• Volume varies from a few readings per day to thousands
• Business rules cross-check sensor readings to make decisions.• Time alignment issues – big challenge for traditional programing languages• Data Conversion issues• Pre-processing spent aligning data for comparisons
Period Data Type
19
The Period data consists of a BEGIN and END pair of DATES or TIMESTAMPS
Period Data Type is key to operating with sensor data and performing analytics using SET procession (SQL)
Period helps solve the issue of “sparse data” and “smooth” time-interval data
• Semantic Views deliver time series data using PERIODS
• Self-Joins to deliver prior-next reading Time Gap/Value
Period Intersections provide answers to business questions
Period Data Type & Intersections
20
123 124 120126
2 1 223
10 20
TS0 TS1
123,2,10 123,3,10
TS2 TS3
12
6
,
2,1
0
TS4
126,2,10 124,2,20
TS6
120,2,20
TS9
S1
S3
S2
Intx
123-126, 2-3 , 10-20, TS0 – TS6Event 120-124, 2-2, 20-20, TS7 – TS9
Sensor Event = S1 >120 & S2 >1 & S3 >=10 & Duration > 30 Seconds
Solution built using Semantic Views to deliver point-in-time sensor readings as Periods. Intersections then deliver common points that can be
analyzed for rules. Recursive SQL is used to string continuous events into one
124,1,20
TS7TS5
124,1,2
0
123,2,10
TS8
Period Data Type & Intersections
21
Semantic View converts 3NF sensor data into a new 3NF Sensor set with Periods of time. Self-Joins with PERIOD Intersections deliver common points that can be analyzed for rules. Self-Join repeated for “n” sensors. Final step applies recursive SQL to string continuous events into a
single row in the EVENT Fact table
3NF Point-in-time Sensor Data Period Sensor Data
Period Sensor Self Join
P_INTERSECT
Period Sensor Intersections of
Interest
Repeat for N Sensors using Derived Tables in a single SQL statement
Sensor EventFact Table
Semantic View
DeriveEvent Dimension
Sensor Events: • Which Sensors, what values, start and end, duration in seconds. • Enough context to drill into detailed Sensor data and plot in Tableau for deeper analysis of an event
Generic Semantic View examples to Deliver Sensor Data with PERIODS
22
Generic Semantic View examples to Deliver Sensor Data with PERIODS
23
Thank You
Questions/Comments
Email:
Follow Me
Twitter @
Rate This Session #
with the PARTNERS Mobile App
Remember To Share Your Virtual Passes
bruce_baum
156
24