on the edge: analytics for the modern enterprise (analyst comments)
TRANSCRIPT
2
Analytics on the Edge: Connected Analytics for the Modern Enterprise
Analyst commentary
August, 2015Mark MadsenThird Nature@markmadsen
Copyright Third Nature, Inc.
Order Entry
Order Database
Customer Service
Interface Program
Inventory Database
Distribution
Interface Program
Receivables Database
Accounts Receivable
Data Warehouse
Analysts & users
Someone else always wants to use your data…
Copyright Third Nature, Inc.
The data warehouse. More of a CostCo, really. Came about in a pre‐event streaming market
Copyright Third Nature, Inc.
Core idea: centralize: that solves all problems!
But there are tradeoffs:
Creates bottlenecks
Causes scale problems
Enforces a single model
Injects latency
Copyright Third Nature, Inc.
New terminology: Data is the new oil
Copyright Third Nature, Inc.
New terminology: Data Exhaust
Copyright Third Nature, Inc.
New terminology: the data lake
Copyright Third Nature, Inc.
There is a different assumption with these terms
“Data isn’t something we make, it’s just there.”
But the prevailing model for management and use is still “centralize”
Copyright Third Nature, Inc.
There is a different assumption with these terms
“Data isn’t something we make, it’s just there.”
But the prevailing model for management and use is still “centralize”
This won’t work
Copyright Third Nature, Inc.
What happens when (a) there are only messages and no database? and (b) machines are actors in the system?
Order Entry
Order Database
Customer Service
Interface Program
Inventory Database
Distribution
Interface Program
Receivables Database
Accounts Receivable
Data Warehouse
Analysts & users
Copyright Third Nature, Inc.
In a mostly‐connected world, events occur in different time frames, follow different cycles of use
13Source: Noumenal
Disconnected
Milliseconds Minutes Hours+
Copyright Third Nature, Inc.
In a mostly‐connected world, events occur in different time frames, follow different cycles of use
14Source: Noumenal
DisconnectedEvery event is
persisted for some period of time before
it is forgotten or forwarded
Milliseconds Minutes Hours+
Copyright Third Nature, Inc.
In a mostly‐connected world, events occur in different time frames, follow different cycles of use
15Source: Noumenal
Disconnected
Local context and control,
local decisions, local latency
Milliseconds Minutes Hours+
Copyright Third Nature, Inc.
In a mostly‐connected world, events occur in different time frames, follow different cycles of use
16
Disconnected
Source: NoumenalMilliseconds Minutes Hours+
Bigger context, likely correlated, more complex rules, external monitoring
Copyright Third Nature, Inc.
In a mostly‐connected world, events occur in different time frames, follow different cycles of use
17
Disconnected
Source: NoumenalMilliseconds Minutes Hours+
Broad context, human intervention, diagnosis and analytical tasks that have to be coordinated.
Copyright Third Nature, Inc.
In a mostly‐connected world, events occur in different time frames, follow different cycles of use
18
Disconnected
Source: NoumenalMilliseconds Minutes Hours+
Data lives in multiple places, at multiple levels of detail, for differing durations. Unlikely to all be in one place.Nor should it be.
Copyright Third Nature, Inc.
Flowing Persisted
Sliding window of “now”
Persisted but not yet loaded into a platform
Queryable history
Managed history
Streaming isn’t either‐or, it’s part of IT architecture
A DB can get you to within minutes (at large scale) but it won’t be easy or cheap, mainly lives in the realm of history
Event streams, in-memstores, CEP streaming SQL can be used for these
Real time monitoring doesn’t use only real time data: windows, restarts, detecting deviation, so the above boundaries are crossed.
ESB Cache/Queue Database / platform
Copyright Third Nature, Inc.
Stream
If you want to do realtime and still manage your data effectively then you need to rethink data architecture
Collect Refine Manage Deliver
Flowing Managed historyPersisted
Microservices MetadataMetadata & reuse?
Flow, persisted, managed define different access, processing, storage and retrieval requirements
Copyright Third Nature, Inc.
Rethinking software and data architectures is required. Enterprise IT needs to be renovated.
Copyright Third Nature, Inc.
Image Attributions
Thanks to the people who supplied the images used in this presentation:
refinery‐hdr.jpg ‐ http://www.flickr.com/photos/vermininc/2477872191/
Building demolition ‐ https://www.flickr.com/photos/gregpc/4429888820Streaming environment ‐ http://www.noumenal.com/
Slide 22
Copyright Third Nature, Inc.
About Third Nature
Third Nature is a research and consulting firm focused on new and emerging technology and practices in analytics, business intelligence, and performance management. If your question is related to data, analytics, information strategy and technology infrastructure then you‘re at the right place.
Our goal is to help companies take advantage of information‐driven management practices and applications. We offer education, consulting and research services to support business and IT organizations as well as technology vendors.
We fill the gap between what the industry analyst firms cover and what IT needs. We specialize in product and technology analysis, so we look at emerging technologies and markets, evaluating technology and hw it is applied rather than vendor market positions.