data models pivot with splunk break out session
TRANSCRIPT
Copyright © 2014 Splunk Inc.
DATA MODELS
During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective
owners. ©2013 Splunk Inc. All rights reserved.
Legal Notices
2
Search is hard
Analytics Big PictureBuild complex reports without the search language
Provides more meaningful representation of underlying raw machine data
Acceleration technology delivers up to 1000x faster analytics over Splunk 5
4
Pivot
Data Model
Analytics Store
Operational Intelligence Across the Enterprise
IT professionalCreate and share data modelsAccelerate data models and custom searches with the analytics storeCreate reports with pivot
Developer AnalystLeverage data models to abstract dataLeverage pivot in custom apps
Create reports using pivot based on data models created by IT
PivotData Model
Raw Data
AnalyticsStore
[10/11/12 18:57:04 UTC] 000000b0 PolicyService E
Pivot is a query builder.
Demo
Data Models 101
Data set
Source
Source
Source
Sourcetype
Success
Failure
Warning
Data set
Business divisionSource
Source
Business divisionSource
Source
Common model
Technology 1
Technology 2
Technology 3
Context
sourcetype=access_combined source = "/home/ssorkin/banner_access.log.2013.6.gz"
| eval unique=(uid + useragent) | stats dc(unique) by os_name
| rename dc(unique) as "Unique Visitors" os_name as "Operating System"
search and filter | munge | report | clean-up
Splunk Search Language
Hurdles
Simple searches easy… Multi-stage munging/reporting is hard!
Need to understand data’s structure to construct search
Non-technical users may not have data source domain knowledge
Splunk admins do not have end-user search context
index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>\d\.\d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as *
Data Model GoalsMake it easy to share/reuse domain knowledge
Admins/power users build data models
Non-technical users interact with data via pivot UI
Data Models 101
What is a Data Model?A data model is a search-time mapping of data onto a hierarchical structure
Encapsulate the knowledge needed to build a searchPivot reports are build on top of data modelsData-independent
Screenshot here
A Data Model is a Collection of Objects
Screenshot here
Objects Have Constraints and Attributes
Screenshot here
Child Objects Inherit Constraints and Attributes
Screenshot here
Child Objects Inherit Constraints and Attributes
Building Data Models
Three Root Object TypesEvent
– Maps to Splunk events – Requires constraints
and attributes
Three Root Object TypesEvent
– Maps to Splunk events – Requires constraints
and attributesSearch
– Maps to arbitrary Splunk search (may include generating, transforming and reporting search commands)
– Requires search string attributes• Transaction
– Maps to groups of Splunk events or groups of Splunk search results
– Requires objects to group, fields/ conditions to group by, and attributes
Three Root Object TypesEvent
– Maps to Splunk events – Requires constraints
and attributesSearch
– Maps to arbitrary Splunk search (may include generating, transforming and reporting search commands)
Requires search string attributesTransaction
– Maps to groups of Splunk events or groups of Splunk search results
– Requires objects to group, fields/ conditions to group by, and attributes
Object AttributesAuto-extracted – default and pre-defined fieldsEval expression – a new field based on an expression that you defineLookup – leverage an existing lookup tableRegular expression – extract a new field based on regexGeo IP – add geolocation fields such as latitude, longitude, country, etc.
Object AttributesSet field types
Configure various flagsNote: Child object configuration can differ from parent
Demo
Data Model Builder UI
Best PracticesUse event objects as often as possible
– Benefit from data model acceleration
Resist the urge to use search objects instead of event objects!!– Event based searches can be optimized better
Minimize object hierarchy depth when possible– Constraint based filtering is less efficient deeper down the tree
Event object with deepest tree (and most matching results) first– Model-wide acceleration only for first event object and its descendants
Warnings!Object constraints and attributes cannot contain pipes or subsearches
A transaction object requires at least one event or search object in the data model
Lookups used in attributes must be globally visible (or at least visible to the app using the data model)
No versioning on data models (and objects)!
From Data Models to Reports
Using the UISubhead
Count of http_success events, split by useragent
events
fields
Under the Hood: Object Search String Generation
Syntax:<constraints search> | <my attribute definitions>
Example:sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
Under the Hood: Object Search String Generation
Syntax:<base search> | <my attribute definitions>
Example:_time=* host=* source=* sourcetype=* uri=* status<600 clientip=* referer=* useragent=* (sourcetype=access_* OR source=*.log) | eval userid=clientip | stats first(_time) as earliest, last(_time) as latest, list(uri_path) as uri_list by userid| earliest=* latest=* uri_list=*
Under the Hood: Object Search String Generation
Syntax:<objects to group search> | transaction <group by fields> <group by params> | <my attribute definitions>
Example:sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* | transaction clientip useragent | eval landingpage=mvindex(uri_path,1) | eval exitpage=mvindex(uri_path,-1)
Under the Hood: Object Search String Generation
Syntax:<parent object search> | search <my constraints> | <my attribute definitions>
Example:sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* status=2* | <my attribute definitions>
Using the Splunk Search Language| datamodel <modelname> <objectID> search
Example:| datamodel WebIntelligence HTTP_Request search
Behind the scenes:sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
Under the hood: Pivot Search String GenerationExample:(sourcetype=access_* OR sourcetype=iis*) status=2* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Sucess" by ”useragent" | sort limit=0 "useragent" | fields - _span | fields "useragent" "Count of HTTP_Success"| fillnull "Count of HTTP_Success" | fields "useragent" *
Using the Splunk Search Language| pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …]
Example:| pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW status AS "status" SORT 0 status
Behind the scenes:sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Request" by "status" | sort limit=0 "status" | fields - _span | fields "status", "Count of HTTP_Request" | fillnull "Count of HTTP_Request" | fields "status" *
Warnings| datamodel and | pivot are generating commands – They must be at the beginning of the search string
Use objectIDs NOT user-visible object names
DemoBuilding a report from a data
model
Managing Data Models
Data Model on DiskEach data model is a separate JSON file Lives in <myapp>/local/data/models(or <myapp>/default/data/models for pre-installed models)Has associated conf stanzas and metadata
Editing Data Model JSONAt your own risk!
Models edited via the UI are validatedManually edited data models: NOT SUPPORTEDException: installing a new model by adding the file to <myapp>/<local OR default>/data/models is probably okay
Deleting a Data Model
Use the UI for appropriate cleanupPotential for bad state if manually deleting model on disk
Interacting With a Data Model
Use data model builder and pivot UI – safest option!Use REST API – for developers (see docs for details)
Use | datamodel and | pivot Splunk search commands
PermissionsData models have permissions just like other Splunk objectsEdit permissions through the UI
Data Model Acceleration
Run a pivot report
Poll: are there new accelerated
models?
Turn on acceleration via
UISetting written to conf file
Kick off collection
Acceleration
Kick off ad-hoc acceleration and run search
Run search using on-disk acceleration
Admin or power user
Backend magic
Non-technical userNo acceleration
Model-Wide Acceleration
Pivot search: | tstats count AS "Count of HTTP_Success" from datamodel="WebIntelligence" where (nodename="HTTP_Request") (nodename="HTTP_Request.HTTP_Success") prestats=true | stats count AS "Count of HTTP_Success”
Only accelerates first event-based object and descendants
Does not accelerate search and transaction-based objects
Ad-Hoc Object Acceleration
Kick off acceleration on pivot page (re) load for non-accelerated models and search/transaction objects
Amortize cost of ad-hoc acceleration over repeated pivoting on same object
Pivot search:| tstats count AS "Count of HTTP_Success" from sid=1379116434.663 prestats=true | stats count AS "Count of HTTP_Success”
Acceleration DisclaimersWorks with search-head pooling – we collect on indexersCannot edit accelerated models
Thank You