splunk modular inputs / jms messaging module input

Splunk Modular Inputs

Damien DallimoreDeveloper Evangelist

Copyright © 2013, Splunk Inc.2

Modular Inputs• Extend the Splunk framework to define a custom input capability, just like the standard inputs you

are familiar with (TCP/UDP/File etc…)

• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally integrated first class citizen objects in Splunk

• Users interactively create and update your custom inputs using Splunk manager, just as they do for native inputs. When deploying without a UI , you push out the inputs.conf file.

• All the properties are fully manageable via the REST API

• Version 5.0 +


What about scripted inputs ?• Very loosely coupled to Splunk• No standard configuration/schema framework• No standard validation framework• No standard lifecycle management• Need to use “hacks” to make them running persistently• Not really integrated with the REST API• Logging not integrating with standard Splunk logs

BUT

• Their simplicity and loose coupling make them very rapid to develop• Choose the right tool for the job


Diagram of Mod Input lifecycle

SplunkD

Init / Request Scheme

Mod Input

Return Scheme

External Validate

Confirm Validation

Execute

XML

XML

XML

Stream ResultsText / XML

Validation Code & Error Msg

$SPLUNK_HOME/var/log/splunk/splunkd.log

logging


Scheme XML


Input XML$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config myscheme mystanza


Manage Mod Inputs via REST API


A few other technical features• Validation

• External mode or via REST create/edit

• Run Mode• single or multiple instance

• Checkpoint directory• So your modular input can maintain state

• Streaming Mode• Text or XML• XML streaming has more syntactic sugar for meta data, event breaking

• Architecture specific scripts• Splunk auto magically chooses the correct runtime script.


How are Mod Inputs going to help us

• We need to make it easy as possible to develop modular inputs , frameworks and tools

• Sometimes the greatest battle is just getting the data in , modular inputs are a great tool in our armory.

• Bundle Modular Inputs in with the core product (DB, JMX, SNMP, JMS etc…)

• We need to make it easy to search for, install and configure these “data connectors”


Developing• My preference is to use Python, however any language can be used.

• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro

• There is a certain amount of “plumbing” to put in place , so I like utilitys that take care of this for you, so you can just focus on the business logic.

• I created utilitys to allow developers to rapidly create Modular Inputs in Java and Python• https://github.com/damiendallimore/SplunkModularInputsJavaFramework• https://github.com/damiendallimore/SplunkModularInputsPythonFramework• HelloWorld examples to get you started

• Java -> JMS Messaging Modular Input , on Splunkbase• Python -> SNMP Modular Input , soon to be released

http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro

http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro

https://github.com/damiendallimore/SplunkModularInputsJavaFramework

https://github.com/damiendallimore/SplunkModularInputsJavaFramework

https://github.com/damiendallimore/SplunkModularInputsPythonFramework

https://github.com/damiendallimore/SplunkModularInputsPythonFramework


Mod Inputs on Splunkbase


Messaging

• Message Oriented Middleware (MOM) infrastructures facilitate the sending/receiving of messages between distributed systems

• Topics (publish/subscribe) and Queues (point to point)

• The glue that stitches heterogeneous enterprise computing environments together

• Represents a massive source of machine data that can be fed into Splunk to derive operational visibility into your messaging environment and the various systems and applications that are communicating via MOM


Building a Splunk Messaging Solution

• There has been considerable demand for functionality in Splunk to index messages from queues/topics

• Ad hoc, proprietary, roll your own solutions were the only way

• I wanted to develop an integrated mechanism to allow Splunk users to connect to their MOM and index their messages

• Modular Inputs provided the perfect platform to build a messaging solution


JMS Messaging Modular Input• JMS is simply a messaging interface that abstracts your underlying MOM provider

implementation• Most MOM vendors support JMS• So this allowed for creating 1 single modular input that can index messages from :

• MQ Series / Websphere MQ• Tibco EMS• ActiveMQ• HornetQ• RabbitMQ• SonicMQ• JBoss Messaging• Weblogic JMS• Native JMS• StormMQ• MSMQ (with a bit of stuffing around)• Etc…

• Simple to install : download from Splunkbase, drop in your apps directory, restart Splunk


Key Features• Known to work with all aforementioned Messaging platforms• Should work against any MOM platform with a JMS provider• Runs on all supported Splunk platforms• Consume messages from Topics and Queues• Browse Queues (if you don’t want to consume the messages) and just Splunk

queue stats• Messages header, properties and body indexed in Splunk in simple key/value

pairs• Can plug in your own message handler if you require customized processing of

the message body• Authentication and SSL support• Scales horizontally if you require large volume message consumption


JMS input fully integrated into Splunk


Add a new queue/topic input


Configure the properties to connect


Get instant operational visibility

DemosJMS (ActiveMQ , Websphere MQ)SNMP


Contact me

Email : [email protected] : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimore Blogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com

http://blogs.splunk.com/dev

http://dev.splunk.com/

http://dev.splunk.com/

splunk modular inputs / jms messaging module input

Technology