splunk's api how we built it

Agenda

Why we built Splunk’s API

How we built Splunk’s API

What is Splunk

A product for handling REALLY large and varied sets of evented data

What is Splunk

From megabytes to hundreds of terabytes daily

What is Splunk

It is highly scalable and distributed

What is Splunk

Useful in many domains: IT/Ops/DevOps, security, healthcare, financial,IOT/Devices

7

Splunk storage Other Big Data stores

Developer Pla6orm

Data collecUon and indexing

Report and analyze

Custom dashboards

Monitor and alert

Ad hoc search

Why – Product need

Need to build a Splunk UI that surfaces all of Splunk’s capabiliUes

Why – Customer need

Need to allow customers to integrate their applicaUons and scripts with Splunk

Why – Cost

Less code to maintain, a single source of truth

Why – Reach

HTTP is ubiquitous, every plaZorm has a client

The arch

12

Server Tech Stack

13

C/C++

Python – Cherry PI Python – Django

Javascript Node

Client Tech Stack

14

Javascript Node Python Ruby PHP C#

The API

15

Log directly to Splunk via HTTP

Run historical and real-‐Ume searches

What can you do with Splunk’s API?

16

Search Manage

Add/Delete Users

ReporUng/Alerts

Manage Inputs

ConfiguraUon

Index

Login to a Splunk instance and get a session token

Auth

The API design

17

Service

Categories

Endpoints

Endpoints

The API design -‐ Categories

18

The API design – Endpoints

19

The API design – Endpoints

20

Responses -‐ Feeds and Hypermedia <entry xmlns="hjp://www.w3.org/2005/Atom" xmlns:s="hjp://dev.splunk.com/ns/rest" xmlns:opensearch="hjp://a9.com/-‐/spec/opensearch/1.1/"> <Utle>search index</Utle> <id>hjps://localhost:8089/services/search/jobs/mysearch_02151949</id> <updated>2011-‐07-‐07T20:49:58.000-‐07:00</updated> <link href="/services/search/jobs/mysearch_02151949" rel="alternate"/> <published>2011-‐07-‐07T20:49:57.000-‐07:00</published> <link href="/services/search/jobs/mysearch_02151949/search.log" rel="search.log"/> <link href="/services/search/jobs/mysearch_02151949/events" rel="events"/> <link href="/services/search/jobs/mysearch_02151949/results" rel="results"/> <link href="/services/search/jobs/mysearch_02151949/results_preview" rel="results_preview"/> <link href="/services/search/jobs/mysearch_02151949/Umeline" rel="Umeline"/> <link href="/services/search/jobs/mysearch_02151949/summary" rel="summary"/> <link href="/services/search/jobs/mysearch_02151949/control" rel="control"/> </entry>

21

Auth

22

HTTP Basic Token based LDAP/AD

Cookie based

Auth – HTTP Basic

23

curl -‐k -‐u admin:changeme hjps://localhost:8089/services/auth/login -‐d username="admin" -‐d password="changeme"

Auth – Splunk Token

24

curl -‐k -‐H "AuthorizaUon: Splunk SfH2D^zvPyLu^mO61C9kWtB7TOuQs0i9oSzh4lD7ho7Gvw26I61VYRjXkgjLQlJDJ0hER^q^A6v0BHYiKNba^CMbOmC63frGCrDqr2Zt" hjps://localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get

Search – Oneshot – Get me results!

25

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot"

Search – Oneshot – Get me results in json

26

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json"

Search – Oneshot – Get me json columns

27

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"

Search – Oneshot – Get me json rows

28

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5" -‐d exec_mode="oneshot" –d output_mode="json_cols"

Search – Blocking – Wait Ull done!

29

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 5” -‐d exec_mode=”blocking” output_mode="json” | python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool

Search – List search jobs

30

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d output_mode="json" -‐-‐get | python -‐mjson.tool

Search – Normal– Run in the background

31

curl -‐u admin:changeme -‐k hjps://localhost:8089/services/search/jobs -‐d search="search sourcetype=sysmon | head 10000" -‐d exec_mode="normal" output_mode="json"| python -‐mjson.tool curl -‐u admin:changeme /services/search/jobs/{sid}/results -‐d output_mode="json" –get | python -‐mjson.tool

Search -‐ Export

32

curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/search/search/jobs/export -‐d search="search index%3D_internal | head 100000" -‐d output_mode="raw"

Search – Export REALTIME

33

curl -‐k -‐u admin:changeme hjps://localhost:8089/servicesNS/admin/search/search/jobs/export -‐d search="search index%3D_internal" -‐d output_mode="raw" earliest_Ume="rt-‐1m" latest_Ume="rt"

splunk's api how we built it

Data & Analytics