MySQL Auditusing percona audit plugin & ELK

About me

dba.kim@gmail.com

Object3

We need - log to MySQL connection information - log to MySQL query log - retrieve these logs using various search conditions - download search results

Architecture

Percona Audit Plugin

4

DB Servers ELK Server

logstash elasticsearch

Audit PluginWhich audit plugin will we use? 5

Percona Audit Plugin MariaDB Audit Plugin

- support various log format(csv, json, xml)

- most of parameter can’t modify online

- support various filtering condition (user, query type : ddl, dml, dcl)- most of parameters can modify online

- stability issue(db crash)https://mariadb.atlassian.net/browse/MDEV-5145http://mysql.az/audit-mysql-no-just-crash-it/

- easy install & config - can log at extra log file or syslog file

We chose Percona Audit Plugin

6

Percona Audit Plugin Install & Config6

mysql>install plugin audit_log soname ‘audit_log.so’;

Ref URL : https://www.percona.com/doc/percona-server/5.5/management/audit_log_plugin.html

$/etc/init.d/mysql restart

$vi my.cnf# Server Audit audit_log_format = JSONaudit_log_policy = ALLaudit_log_syslog_facility = LOG_LOCAL1audit_log_handler = SYSLOG

1. Percona Audit Plugin Install (over MySQL Ver 5.5.39) – on DB servers

2. Parameter configuration

3. MySQL restart

rsyslog config7

1. rsyslog.conf – on DB servers$ vi /etc/rsyslog.conf# mysql logginglocal1.* @10.xxx.xxx.xxx # ELK server ip

2. rsyslog restart

$ /etc/init.d/rsyslog restart

3. confirm log messages on syslog file$cat messagesJan 29 15:08:30 testdbsvr01 percona-audit: {"audit_record":{"name":"Query","record":"3683778651_1970-01-01T00:00:00","timestamp":"2016-01-29T06:08:30 UTC","command_class":"select","connection_id":"455338789","status":0,"sqltext":"SELECT * FROM test_table WHERE status='Done'","user":"app[app] @ [10.xxx.xxx.xxx]","host":"","os_user":"","ip":"10.xxx.xxx.xxx"}}

8

rsyslog-ng config

1. rsyslog.conf – on ELK servers

$ vi /etc/syslog-ng/syslog-ng.conf

source s_sys { file ("/proc/kmsg" program_override("kernel: ")); unix-stream ("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); ## uncomment this line};

destination d_myaudit { file("/var/log/myaudit/myaudit.${HOST}.${YEAR}-${MONTH}-${DAY}.log" create-dirs(yes) dir-perm(0755) perm(0644) ); };filter f_myaudit { facility(local1); };log { source(s_sys); filter(f_myaudit); destination(d_myaudit); };

2. syslog-ng restart

$ /etc/init.d/syslog-ng restart

Install ELKElasticsearch 9

1. elasticsearch install – on ELK server

$ yum install elasticsearch

2. configuration

$ vi /etc/elasticsearch/elasticsearch.ymlcluster.name : my_cluster # cluster namenode.name : my_node01 # node namenetwork.host : 10.xxx.xxx.xxx # server’s ip

You can download from https://www.elastic.co/downloads/elasticsearch.

It needs a recent version of java before install elasticsearch.

Install ELKElasticsearch 10

3. start elasticsearch

$/etc/init.d/elasticsearch start

4. Head plugin install

$cd /usr/share/elasticsearch/bin$./plugin install mobz/elasticsearch-head

5. Check plugin installhttp://10.xxx.xxx.xxx:9200/_plugin/head/

1111

Install ELKLogstash 11

1. Install Logstash – on ELK server

$ rpm –ivh logstash-2.1.1-1.noarch.rpm

2-1. Configure ( input plugin )$ sudo vi /etc/logstash/conf.d/01-myaudit-input.-confinput { file { type => "myaudit" path => ["/var/log/myaudit/*.log"] start_position => "beginning" codec => 'json' }}

You can download from https://www.elastic.co/downloads/logstash

12

Install ELKLogstash

2-2. Configure ( filter plugin )

$ sudo vi /etc/logstash/conf.d/10-myaudit.conffilter { grok { match => { "message" => "%{SYSLOGTIMESTAMP:sys_timestamp}%{SPACE}%{HOSTNAME:host_name}%{SPACE} percona-audit: %{GREEDYDATA:json_data}"} } json { source => "json_data" }}

13

Install ELKLogstash

2-3. Configure ( output plugin )

$ sudo vi /etc/logstash/conf.d/30-elasticsearch-output.confoutput { elasticsearch { hosts => "10.xxx.xxx.xxx"}}

3. Start logstash

$ sudo /etc/init.d/logstash start

14

Install ELKKibana 14

1. Kibana install – on ELK server

$ tar –xvf kibana-4.3.1-linux-x64.tar.gz 2. configuration

$ vi ./config/kibana.ymlhost: “10.xxx.xxx.xxx” # kibana server ipelasticsearch_url: “http://10.xxx.xxx.xxx:9200” # elasticsearch server ip

4. Check

You can download from https://www.elastic.co/downloads/kibana

http://10.xxx.xxx.xxx:5601

3. Start Kibana

$ ./bin/kibana

It needs a space between “host:” and IP.

15

Install ELKKibana

5. Config an index pattern

16

Make Kibana Dashboard

http://10.xxx.xxx.xxx:5601

Search condition create(1/3) 17

Make Kibana Dashboard

1

2

Search condition create(2/3) 18

Make Kibana Dashboard

3

4

19Search condition create(3/3)Make Kibana Dashboard

5

6

20

1

Make Graph: line chart(1/6)Make Kibana Dashboard

21

2

Make Kibana DashboardMake Graph: line chart(2/6)

22

3

4

Make Kibana DashboardMake Graph: line chart(3/6)

23

5

6

Make Kibana DashboardMake Graph: line chart(4/6)

24

7

8

Make Kibana DashboardMake Graph: line chart(5/6)

25

9

Make Kibana DashboardMake Graph: line chart(6/6)

26

Make Kibana DashboardMake Graph: Pie chart(1/4)

1

27

Make Kibana DashboardMake Graph: Pie chart(2/4)

2

28

Make Kibana DashboardMake Graph: Pie chart(3/4)

3

4

29

Make Kibana DashboardMake Graph: Pie chart(4/4)

5

30

Make Kibana DashboardMarkdown widget(1/3)

1

31

Make Kibana DashboardMarkdown widget(2/3)

1

###Menu: // label, the number of “#” determines a size of character.[Main]: //label(/#dashboard/Main): // link dashboard. “Main” is name of dashboard. You have to make dashboard before markdown widget.

2

32

Make Kibana DashboardMarkdown widget(3/3)

3

33Make DashboardMake Kibana Dashboard

1

34

2

Make DashboardMake Kibana Dashboard

35

3

4

Make DashboardMake Kibana Dashboard

Dashboard-sample36

Markdown widgetLine chart

Search condition

Dashboard-sample37

Markdown widgetPie chart

Elasticsearch Plugin – Head(1/2)38

http://10.xxx.xxx.xxx:9200/_plugin/head/

1

2

3

4

search query

Elasticsearch Plugin – Head(2/2)39

{

"_source": [

"@timestamp",

"host_name", "audit_record.user", "audit_record.sqltext", "audit_record.name" ],

"sort": { "@timestamp": "asc" },

"size": 100,

"query": {

"bool": {

"must": [

{

"range": {

"@timestamp": {

"gte": "2016-01-16T00:00:00.000",

"lte": "2016-01-16T02:59:59.999" }

} },

{"term": { "host_name": “host_name123" } },

{"term": { "audit_record.name": "query"} }

] } }}

Search condition of string type uses a lower character only.ex) “audit_record.name”:”Query” (x) “audit_record.name”:”query” (o)

Thank You